Close
0%
0%

Luxonis DepthAI

Bringing the power of the Movidius Myriad X to Raspberry Pi
(and your design!)

Public Chat
Similar projects worth following
The Myriad X is a very powerful vision processor capable of doing real time neural inference at over 30FPS, stereo disparity depth on up to 3 pairs of cameras, and general computer vision functions - all at very low DC power. Up to 4 TOPS in a small, low DC-consumption part!

At Luxonis we're aiming to unleash this power by making a Myriad X System on Module which allows embedding the power of the Myriad X into your own products.

And to boot, we’re making a carrier board with the same form-factor and connectors as a Raspberry Pi, which will hold both modules:
a. The Raspberry Pi Compute Module (3B+)
b. The Luxonis Myriad X System on Module

This allows a couple things:
1. The video data path now skips the Pi, eliminating that CPU use (which is a LOT).
2. The hardware stereo-image depth capability of the Myriad X can now be used.
3. An estimated ~5x improvement on MobileNet-SSD object detection, as a result of the Raspberry Pi CPU no longer limiting the X.

So the effort that prompted us making the DepthAI solution is actually Commute Guardian (check it out at commuteguardian.com).  So we have a lot of that effort (prototyping, results, etc.) mixed in here.

We have to do all (and more) of the work for the DepthAI for that end-goal (which is itself to save bikers' lives).  And so we figured, why not share the general underpinnings to Commute Guardian, here, with Hackaday - so others can benefit from the core work we're doing on Commute Guardian.

For more information on what we're thinking on the DepthAI, stay tuned here, and/or check out https://discuss.luxonis.com/ to give feedback/feature-requests/etc. as we progress along making the device.

  • 1 × Intel Myriad X Vision/AI Processor
  • 1 × CM3B+ Raspberry Pi Compute Module 3B+
  • 2 × Camera Connectors TBD, comments welcome!

  • Starting to Calibrate and Experiment on Disparity Depth

    Brandon2 days ago 0 comments

    Hi everyone,

    So all the hardware updates so far have been the work of our lead hardware engineer, Brian.  Today we'd like to introduce a new addition to our team, Martin, who's expertise is in firmware (and software) and is taking on depth work on the Myriad X to start with (and then will be tackling all sorts of other Myriad X firmware and software).

    So Martin's made some short work on getting the depth performance to be better, finding and solving a couple bugs, and experimenting with some depth processing techniques to see about improving the depth results (which will eventually, if they work well, will be implemented onto the SHAVES of the Myriad X).

    And a big thanks to David and Michael at https://petronics.io/ for pointing us in the right direction here!

    Here's the depth prior to calibration:

    It turns out we had been using non-calibrated depth as result of a bug in loading the calibration that was found and fixed.

    And here is with the bug fix and calibration applied:

    And here's with some edge detection on the mono and smoothing like the Intel D435 does:

    And these are just initial guesstimates at the coefficients to use for smoothing, so they're probably not ideal for our setup.  

    We quickly tried a bit less smoothing and that seems even better:

    So we're still tweaking and experimenting, but we figured we'd update as these are starting to look about as good as the results from the D435 from our CommuteGuardian prototype (here).

    And for comparison with that video, below are some results from median filtering (first video) and also mono camera-based edge detection and smoothing (second video).  The edge detection isn't synced properly yet, so you can see ghosting as the mono (grayscale) video feed gets out of sync w/ the depth feed when there's motion.

    It's worth noting that this is depth on sensors that are pretty close-in, so that's why it quickly progresses to being towards the edge of the perceivable range of the disparity depth.  Here's the board this is running on, which has the global-shutter (disparity depth) cameras spaced at ~34mm.

    And on our DepthAI for Raspberry Pi the spacing will be much larger, which will a lot further-out depth sensing (likely around 12-15 meters), with spacing of 110mm between the cameras instead.

    More updates to come!

    Cheers,

    The Luxonis team!

  • Myriad X Module Delayed, First Carrier Board Arrived

    Brandon2 days ago 1 comment

      Hi everyone!

      So I want to start out this post that on every other project with MacroFab we've been quite pleased with MacroFab - so we do think this is an exception and a one-time sort of thing with them.

      So we got bad news yesterday that our Myriad X module, which we ordered back on June 26th, had gotten lost in their system.   Worse, it was lost in such a way that the automation was telling all their management and engineers (and us) that everything was in-progress and on-time to ship on July 15th.  So when we kept asking (as we're super excited to get these back), they (truthfully) kept telling us that all was good - as that's what their system was telling them.  (Such a thing actually belies an impressive amount of automation on the part of MacroFab, actually, that there's such trust in the automation, and that such extensive automation exists.)

      So anyways, on July 10th when we checked in once again, the following anomaly popped up this time:

      Notice that the start date is set for 8 days after the ship date.  Doh!

      This prompted MacroFab to dig in more... it was the first sign that something was amiss.  And in digging in, they discovered that there was a corner-case bug with their recent upgrade of their automation software, and our order and 2 orders had fallen into that corner.  :-)

      So what does this mean for us?  We thought the 3-week countdown to having the Myriad X module started on June 26th.  Nope.  Because of this bug, it actually starts today!  So we're looking at August 8th for our first modules to come in.

      A bummer, for sure, but this happens in the world of doing things, making things.  And, the good news, is the little breakout board for the module did arrive on time (we ordered them so that they'd arrive same day, which was yesterday):

      So we tested as far as we could on these, but real testing will happen when the Myriad X module that this is designed for gets in around August 8th.  Anyways, they look good so far (they're pretty simple, mainly just a breakout board):

      1. No solder joints looked problematic
      2. Application of 5V was nominal, and the 3v3 regulator turned on and generated 3.3V
      3. The tiny FB chokes on the differential pairs all showed continuity from one side to the other, but also showed isolation from + to - lines of the same diff pair, indicating proper soldering
      4. 5V is present at the input of the 1099 connector and also at both camera connectors, as expected. 

      We also have some stereo depth firmware updates we're going to share soon!  Some promising results!

      Best,

      The Luxonis Team

  • DepthAI for Raspberry Pi Layout In Progess

    Brandon07/10/2019 at 22:45 0 comments

    Hi everyone!

    So we're now in progress laying out our first revision of DepthAI for Raspberry Pi.  This is the one that has a slot for a Raspberry Pi Compute Module (3B+).

    First, pictures of the layout as it's progressing now, and then the brief specs:

    So in short our Myriad X module (the BW1099) pops on the front of the board, the Raspberry Pi Compute Module pops on the back, and the two global-shutter cameras are on front for depth, along with the 12MP color sensor which supports 4K video output (H.264 encoded) to the Raspberry Pi as well.

    And for the full specs:

    • HDMI Connector
    • CM3 Connector
    • LAN9513 for 3x USB 2.0 HUB & Ethernet (10/100)
      • 2x USB 
      • 1x USB slave input to accommodate USB boot config from CMIO (might remove later)
      • 1x hard-wired USB 2.0 lane to Myriad X module
      • 1x Ethernet
    • Standard Raspberry Pi header
    • Display connector (same as RPI 3 B+)
    • Rpi Camera connector (same as RPI 3 B+)
      • This is so you could use the board w/out our image sensors if they want… i.e. this is like ‘embedded NCS2’ version.
    • Sd-card slot
    • Standard 3.5mm audio jack
    • Three onboard cameras:
      • 1 x color; 12 MP: IMX378
      • 2 x grayscale for depth; 1MP each: OV09282
    • BW1099 Connector (this is our Myriad X module, the BW1099)

    Thoughts?  

    We're hoping to sell the whole package in late September for $199 and will be launching a CrowdSupply page soon (to help get the necessary volume order to drive the cost down).

    Cheers,

    The Luxonis Team

    luxonis.com

  • Modular Design Ordered (Including a Test Board)

    Brandon07/02/2019 at 02:50 0 comments

    Hi everyone!

    So we ordered the Modular design for the Movidius Myriad X.  As below it breaks out MIPI, and will be the core of our products.

    BUT!  It also allows you to integrate this into your prototypes and/or products.  Without the hassle of dealing w/ a super-fine-pitch tiny ~400 pin BGA!

    Instead it's an easy Hirose connector that gives you everyone you want, and nothing more.  In a convenient 40mm x 30mm package.

    We do have a question:

    Should we make one of these with a built-in host controller, say an i.MX 8M?

    And oh by the way here's the breakout board, which allows connection to the IMX378 modular sensor and also 2x OV09282 (for stereo disparity depth):

    Cheers,

    The Luxonis Team

  • Module Design Almost Complete

    Brandon05/29/2019 at 16:33 0 comments

    Hi everyone,

    We're making a modular design - and it's almost done.  We'll probably order our first round next week.

    Why?

    • Allows integration of the Myriad X directly into your design.
    • Doesn't require your host have PCIE (USB instead)
    • Allows plug/play Raspberry Pi + Myriad X at 30FPS instead of 8FPS of NCS2, including stereo depth
    • Allows _you_ to make whatever board you want, without having to muck with dealing with the complexity and cost of making a high-density BGA board.

    How?

    So we'll make two carrier boards for this to start, both of which will be open source:

    1. A carrier board for this module with an output USB interface
    2. A carrier board for this module and also for the Raspberry Pi Compute Module

    What?

    • Myriad X with all supporting hardware.  All the carrier board needs to do is provide power and connectivity.
    • All connectivity through single 100-pin connector (DF40C-100DP-0.4V(51))
    • 1 x 4-lane MIPI for high-res image sensor (e.g. 12MP)
    • 2 x 2-lane MIPI for standard-res image sensors (e.g. 1MP)
    • USB interface
    • 40mm x 30mm (~ 1.57" x 1.18")

  • First Boards Work!

    Brandon05/26/2019 at 04:42 2 comments

    Hey everyone!

    So the first boards we designed were an internal development board of sorts.  Initial specifications:

    • Breaks out connections for 5 cameras (including 2 Raspberry Pi connectors)
    • Has SD-Card, emmc, SPI-flash and a bunch of other things we probably don't need, but we figured we'd break out in case we want to develop for them on a future product/functionality (without having to order new boards)
    • Has connectors for our dual-camera module (for disparity-depth)

    The renderings of these are in the previous post... but now we have working boards, so for some pictures!

    So the first thing we did was run hello world via JTAG... and it worked!

    I studied German at one point, so now I like to do 'Hello World' in German instead. ^

    Then, we went ahead (gasp!) and tried out to see if the image sensors all worked properly:

    And they totally did.  Above you can see (uncalibrated) stereo disparity depth and also feature tracking.

    So the next thing we're working on is making a module for the Myriad X... as we (re)discovered how pricey boards of this caliber are when you do low quantities.

    So the module will allow us to do a variety of different form-factors, hosts (e.g. Raspberry Pi Compute Module, Toradex NXP i.MX8 Computer on Module, etc.) that the Myriad X can pop into.

    Cheers!

     - The Luxonis Team

  • First Boards Ordered! Modular Design in Progress

    Brandon04/29/2019 at 18:14 5 comments

    Hey everyone!

    So we did it.  We ORDERED our first boards.  Man, low-volume pricing on tight-tolerance boards is so painful.

    Fortunately we think our kids are crafty, so they won't need a college fund, right?


    We're actually going with a modular approach from what we learned from this, so that we can make various hardware incarnations, which will all leverage the same Myriad X module.

    That way we don't have to give up our kids' college funds every time we decide to change features/form-factor/etc.

    So that's what we're currently working on.  The current plan for the module are 2 x MIPI 4-lane which are usable as 4 x MIPI 2-lane and of course USB and all that jazz.

    The idea is that this allows up to 4x 2-lane cameras, 1x 4-lane camera and 2x 2-lane cameras, or 2x 4-lane cameras.

    Which we think should cover any/all permutations of the boards that this module will go on.  If there's interest, we'll sell the module alone as well.

    Thoughts?

    And as a bonus, here's one of the camera modules, which is a greyscale stereo pair:

  • Hardware Depth and Video Tracking Working

    Brandon03/18/2019 at 16:19 0 comments

    Hey guys,

    So we got hardware depth and video tracking working.  It's not calibrated depth yet (so that's why it doesn't look so great - it's using a unity 3x3 homography matrix).  But it's working!  (Caveat on that, it's still buggy and crashes on startup 9/10 times, but the 1/10 is so satisfying!)

    But to re-iterate, all the calculation shown in the video is being done on the Myriad X (depth calculation and feature tracking).  The host is doing nothing (other than just displaying the data that the Myriad X is streaming, which is optional).  

    The nice part is the Myriad X doesn't even get warm doing this.  And that's with zero heatsink.  Just the chip exposed to ambient air.


    And for more info as to our end goals, check out:
    aipi.io - a Raspberry Pi depth vision + AI carrier board, which is itself a product we thought would be useful to the world, and is an internal stepping stone to:
    commuteguardian.com - the AI bike light to save lives

  • We Have a Logo!

    Brandon03/05/2019 at 05:30 0 comments

    So we've been chasing down parts for our first board run (of 10 prototype/dev.) units, and in the meantime we now have a logo, thanks for our fancy graphic designer:

  • First Board Design | Component Placement Done

    Brandon02/19/2019 at 03:53 3 comments

    Hey Embedded AI Enthusiasts,

    We're excited to share that we just finished component placement and initial routing of our first version of the board.  This one is for initial development, debugging, etc. - and actually doesn't even have a Raspberry Pi slot yet.  It'll primarily be programmed by JTAG and prodded and debugged.

    Anyways, here's a 3D view of it:

    It is, however, the same size as a Raspberry Pi 3.  For the later versions, we'll remove a TON of extra stuff that's on this one - so there'll be more room for the Raspberry Pi CM3B+ module.

    Thoughts?

    Best,

    AiPi Team!

View all 21 project logs

Enjoy this project?

Share

Discussions

psykhon wrote 03/07/2019 at 12:19 point

Hi Brandon, awesome project! 

How hard was to get te myriad x chips? Can you share some info on how do you do it? price?

  Are you sure? yes | no

Tegwyn☠Twmffat wrote 02/01/2019 at 14:53 point

I checked out that link above - I wonder how their larger kit compares to the Jetson TX2 in terms of performance?

I realise performance is not everything and the Intel model zoo is pretty useful. The Nvidia software seems to be a bit behind in that they only have blvc_Googlenet as 'out of the box' solution for detection.

What do you think your price point will be for a single myriad X carrier board, I'm presuming about $100 ?

  Are you sure? yes | no

Brandon wrote 02/01/2019 at 15:02 point

Great question!  So we've actually done a decent amount of stuff on the Tx2 as well.  The Myriad X, in terms of straight neural inference performance (e.g. object detection, semantic segmentation, etc.) is about the same as the Tx2.  The Myriad X neural engine is 1 TOPS, and the Tx2 peaks in ideal conditions at 2 TOPS, but from below, it seems like in most conditions, it's effectively 1 TOPS:

https://devtalk.nvidia.com/default/topic/1024825/cuda-programming-and-performance/jetson-tx2-performance/

But!  If your application is depth vision + neural inference, the Myriad X is equivalent to about 2 or 3 Jetson Tx2, mainly because of the 16 SHAVE cores in the Myriad X, which together can do 6 cameras in 3 pairs of depth streams. 

The neural inference part of the Myriad X is only 1 TOPS of the total 4 TOPS the device an do.  The remaining tops are for image processing functions like depth vision.

So this board won't really even tax the Myriad X, as there will just be one depth stream.  That said, we can use the extra Myriad X 'head room' to run fancier/more-processing-intensive depth calculation on these just 2 cameras - to produce a better set of depth information.

  Are you sure? yes | no

Tegwyn☠Twmffat wrote 01/31/2019 at 22:58 point

Hello Brandon! Does the Myriad X chip get put on the carrier board or does it stay in the USB stick?

If it goes on the board, how many of them?

  Are you sure? yes | no

Brandon wrote 02/01/2019 at 12:37 point

The Myriad X would be directly on the carrier board.  We could make versions with multiple Myriad X, for sure.  Is that of interest?  

These guys did that for their PCIE version:

https://www.crowdsupply.com/up/ai-core-x

I have 2 of those on order, by the way.  They're useful as well, for sure - just a different application, and not applicable for the Pi community (which is what this board should serve).

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates