Close
0%
0%

Luxonis DepthAI

Spatial AI Meets Embedded Systems

Similar projects worth following
The Myriad X is a vision processor capable of doing real time object detection and stereo depth at over 30FPS.

Let's unleash this power!

How?
We're making a Myriad X System on Module (SoM) which allows embedding the power of the Myriad X into your own products, with firmware support for 3D object detection/location.

The module is architected to enable:
1. The video data path to skip the host, eliminating that additional latency and bottlenecking
2. Stereo depth capability of the Myriad X for 3D object position
3. Significant reduction in the CPU load on the host.

We've built a hardware suite around this module, with 3 editions:
- USB3 with Modular Cameras (compatible w/ Yocto, Debian, Ubuntu, Mac OS X, Windows 10)
- RPi HAT with Modular Cameras
- RPi Compute Module with Integrated Cameras

Real-time 3D position of objects, on your favorite embedded platform!

Our Crowd Supply Campaign is Live!

Back us by ordering DepthAI here: https://www.crowdsupply.com/luxonis/depthai

The Why

  • There’s an epidemic in the US of injuries and deaths of people who ride bikes
  • Majority of cases are distracted driving caused by smart phones (social media, texting, e-mailing, etc.)
  • We set out to try to make people safer on bicycles in the US
    • We’re technologists
    • Focused on AI/ML/Embedded
    • So we’re seeing if we can make a technology solution

Commute Guardian

(If you'd like to read more about CommuteGuardian, see here)

DepthAI Platform

  • In prototyping the Commute Guardian, we realized how powerful the combination of Depth and AI is.
  • And we realized that no such embedded platform existed
  • So we built it.  And we're releasing it to the world through a Crowd Supply campaign, here

We want this power to be easily embeddable into products (including our own) in a variety of form-factors (yours and ours).  So we made a System on Module which exposes all the key interfaces through an easy-to-integrate 100-pin connector.  

Unlike existing USB or PCIE Myriad X modules, our DepthAI module exposes 3x MIPI camera connections (1x 4-lane, 2x 2-lane) which allows the Myriad X to received data directly from the camera modules - unburdening the host

The direct MIPI connections to the Myriad X removes the video data path from the host entirely.  And actually this means the Myriad X can operate without a host entirely.  Or it can operate with a host, leaving the host CPU completely unburdened with all the vision and AI work being done entirely on the DepthAI module/Myriad X.

This results in huge efficiency increases (and power reduction) while also reducing latency, increasing overall frame-rate, and allowing hardware blocks which were previously unusable to be leveraged.

Take real-time object detection on the Myriad X interface with the Raspberry Pi 3B+ as an example:

Because of the data path efficiencies of DepthAI vs. using an NCS2, the frame rate increases from 8 FPS to 25FPS.  

And most importantly, using this data path allow utilization of the following Myriad X hardware blocks which are unusable with previous solutions:

This means that DepthAI is a full visual perception module - including 3D perception - and no longer just a neural processor, enabling real-time object localization in physical space, like below, but at 25FPS instead of 3FPS:


And to allow you to use this power right away, and with your favorite OS/platform, we made 3x editions of DepthAI, which serve as both reference designs for integrating the DepthAI module into your own custom hardware and also as ready-to-use platforms that can be used as-is to solve your computer vision problems, as-is.
  1. Raspberry Pi Compute Module Edition - with 3x integrated cameras
  2. Raspberry Pi HAT Edition - with 3x modular cameras
  3. USB3 Edition - compatible with Yocto, Debian, Ubuntu, Mac OS X and Windows 10

All of the above reference designs will be released should our CrowdSupply campaign be successfully funded.  So if you'd like to leverage these designs for your designs, or if you'd like to use this hardware directly, please support our CrowdSupply campaign:

https://www.crowdsupply.com/luxonis/depthai

Development Steps

The above is the result of a lot of background work to get familiar with the Myriad X, and architect and iterate the System on Module definition, form-factor, interfaces, and manufacturability.  Below are some of the steps involved in that process.

The first thing we made was a dev board for ourselves.  The Myriad X is a complicated chip, with a ton of useful functionality... so we wanted a board where we could explore this easily, try out different image sensors, etc.  Here's what that looks like:

BW0235

We made the board with modular camera boards so we could easily test out new image sensors w/out the complexity...

Read more »

  • 1 × Intel Movidius Myriad X Vision/AI Processor
  • 1 × CM3B+ Raspberry Pi Compute Module 3B+
  • 2 × OV9282 Global Shutter camera modules optimized for disparity depth
  • 1 × IMX378 Nice high-resolution 12MP camera module that supports 12MP stills

  • Enclosures and Mounts

    Brandon7 days ago 0 comments
  • Crowd Supply Shipments Received and Community Slack

    Brandon03/02/2020 at 18:59 0 comments

    Hi everyone,

    Just wanted to share the the Crowd Supply shipments have been received, and our fans are up and running with the hardware, with much exciting activity and discussion in our Community Slack channel.

    To join, go to our docs page: https://docs.luxonis.com/ and scroll down to the bottom and click on 'Slack Community' to join.

    We also have a couple new hardware models for sale, and a PoE variant of DepthAI in progress.

    First, the two new models since the Crowd Supply campaign (these will be on the campaign soon):

    1. BW1098OBC - USB3, Onboard cameras:

    2. BW1093 Single camera variant, USB3C.  We're calling this uAI (pronounced 'micro AI').

    You can buy initial versions of these here:

    https://shop.luxonis.com/

    Note that the BW1093 has a temporary heatsink on these units, as the Coronavirus is preventing the CM from being able to make the final version.  So it's just a stick-on one for now, similar to what the Raspberry Pi stick-on heatsinks look like:

    And for the new board we're working on:

    We got requests to be able to deploy DepthAI quite far from any host computer.  And to be able to deploy as many as people want.  Boom, enter PoE.  Now you can deploy DepthAI at up to 328.1 feet (100 meters, for those who have 1m = 3.281 feet memorized, like I do, strangely).

    Below are the renderings of this Myriad X power over Ethernet Design.  This is more or less our initial test design.  Other variants will come after this is validated and proves out all the EE pieces.  

    Our plan is to make an ROS module so that when plugged into the network ROS will auto-discover it and it can be used right away.

    Thoughts?

    Thanks,

    Brandon and the Luxonis team!

  • MVP Complete, Crowd Supply Shipping

    Brandon02/14/2020 at 16:40 0 comments

    Hi all,

    So 1/2 of the Crowd Supply orders shipped to customers yesterday.  And the other half are shipping out today.  

    So guess what?  Our planned shipped date was... drum roll please, February 14th (today)!

    And we have our MVP running:

    So what is this showing?  The dawn of 'spatial AI'.  Where now your embedded system can know what objects are, and where they are in physical space, in real time.  

    It's a capability that straight-up didn't exist at all prior to 2016, even on mainframes.  And even recently, thinking you could do this on an embedded system would be considered science fiction.

    Now you can have this capability, for your product, with this little module:

    And big thanks to the Crowd Supply team for getting these all shipped out on time (and early)!

    Best,

    Brandon & the Luxonis team.

  • PCB Assembly 50% Done

    Brandon01/03/2020 at 18:23 0 comments

    Hi DepthAI fans!

    So the following PCBAs just finished soldered assembly (final assembly is still to be done):
     - DepthAI: USB3 Edition
     - DepthAI: System on Module
     - 4K, 60 Hz Video Modular Color Camera
     
     Below are some pictures of the assembled/soldered boards and them shipping out to the US.
     **DepthAI: USB3 Edition**

     
     **DepthAI: System on Module*

     
     **4K, 60 Hz Video Modular Color Camera**


     
     The following DepthAI variants are undergoing soldered assembly now, with the goal to finish before Chinese New Year starts.
     - DepthAI: RPi Compute Module Edition
     - DepthAI: RPi HAT Edition
     - 720p, 120 Hz Global Shutter Modular Stereo Camera Pair
     
    So this year Chinese New Year is January 18th through February 2nd, where most companies will be offline.  The remainder
    will be offline January 11th through the February 2nd.  

    So our strategy on fulfillment is (and has been) to conclude all the China-involved production prior to January 11th, doing the remainder of the full-system assembly and test in the United States while China is offline to hit our target of DepthAI CrowdSupply shipments by February 14th (to show our love!).  

    It looks like we're still on schedule to hit this, but it going to be close!

    We'll keep everyone updated.  

    Thanks,
    Brandon and the Luxonis Team

  • Tri-Camera USB3C Variant

    Brandon12/05/2019 at 19:00 0 comments

    Hey everyone,

    So out of the model variants we've made/released on our Crowd Supply (here), we've gotten a TON of feedback to have two other variants:

    1. Single-camera, USB interface, and TINY.  (So we made and ordered that, see last update here)
    2. Tri-Camera, USB interface, and human-eye camera spacing.  Allowing easy plug/play onto platforms where the human-eye spacing is sufficient for the stereo baseline.

    So, we made 2 as well, and it's arrived!, and works great!  Pictures below.

    Here's our first check of it (before thermal-padding/gluing the cameras down), just to make sure all the cameras work as intended.  This is also of course not calibrate for depth (as the cameras are flowing by their FFC connections at this point).

    Thoughts?

    Cheers,

    Brandon & the Luxonis Team


  • Single-Camera AI USB3 Camera

    Brandon11/22/2019 at 23:23 0 comments

    Hey Guys,

    So one piece of feedback we've gotten a ton of is 'That whole Depth + AI thing is great, and we're going to use it on several efforts... but, do you have a small single-camera version for AI only without depth?'

    So, we're making it.  And we're making it small:

    So it looks like we're finish the simplified single camera design today (heavily leveraging our existing module) and likely get it ordered tomorrow.  So this means we'll have it in-hand and ready to go probably mid December.
    image.png

    ^ The gray box is the color camera (we haven't bothered to take the time to make a model of it).  

    And the pink part is the thermal interface material (TIM) which connects the Myriad X to the heatsink.  It's the tallest part on the back, intentionally, to make it so this can be mounted easily to existing hardware and heat-sink to it.
    image.png

    To put a scale to this, it's approximately the same dimensions as this (the unit above is 44mm x 30mm, and the module below is 40mm x 30mm):
    image.pngAnd if anyone is interest in this, we do have an initial version that our intern made, which is a two-board solution (which is much larger... hey, it's an intern project!) but is a working system:

    Cheers,

    Brandon and the Luxonis Team!


  • DepthAI Crowd Supply Campaign Live!

    Brandon11/04/2019 at 19:41 0 comments

    Hey everyone,

    Our Crowd Supply campaign is live!  Please help us bring DepthAI to the world by backing our campaign!

    https://www.crowdsupply.com/luxonis/depthai

    Thanks!

    Brandon and the Luxonis Team

  • Crowd Supply Campaign Going Live Monday!

    Brandon11/02/2019 at 03:48 0 comments

    Hi everyone,


    Our Crowd Supply campaign is going live on Monday!


    So if you haven’t done so already, please subscribe on our project page to get notified when the DepthAI campaign starts!


    https://www.crowdsupply.com/luxonis/depthai


    Best,

    Brandon and the Luxonis Team!

  • Preparing for CrowdSupply Campaign Launch

    Brandon10/28/2019 at 15:07 0 comments

    Hi everyone,

    So we've been heads-down preparing for our Crowd Supply campaign launch (hence a bit of lack of updates here lately). 

    In preparation for that, we got our github up, which should the campaign succeed, will host our source design files for both hardware and software.  (If the campaign doesn't succeed, we're going to keep these hardware and software source files internal for our own use only).

    So without further ado, below are some details on the Crowd Supply campaign (which can be found on our github as well, here).

    DepthAI Crowd Supply Campaign

    We are launching a CrowdSupply campaign (https://www.crowdsupply.com/luxonis/depthai) to bring the power of the Myriad X to your design. If the campaign is successful we will be releasing the designs (hardware and software) for the following Myriad X carrier boards here on Github - to enable engineers and makers to build their own solutions, leveraging our source designs/code.

    In addition to the designs shown above (which will be purchase-able through the campaign), we will be releasing an ordered, but not-yet-received version which integrates all 3 cameras onto a single board with a USB3C interface to the host, shown on the bottom right below.

    DepthAI Models

    The Why of DepthAI

    The Myriad X is WAY faster and a TON of features are only accessible when cameras are connected directly to it over MIPI. Take object detection below as an example: DepthAI Models

    So that's 25FPS instead of 8FPS... and that frame-rate is achieved with no host processing AT ALL! (Technically, you can run DepthAI with no host at all - but often you'll want a host so that other business-logic code can be run based on the output of DepthAI... like 'pick only the ripe strawberries')

    There's more to it than just being faster though... the DepthAI platform is engineered from the ground up to enable the original vision of the Myriad X - which is to allow low-power, high-efficiency vision processing including stereo depth, motion estimation, and neural inference. Existing platforms have no MIPI interface and only have a PCIE or USB inferface and so cannot take advantage of a TON of the hardware modules built into the Myriad X which are only (meaningfully) usable with MIPI:

    • Stereo Depth
    • Edge Detection
    • Harris Filtering
    • Warp/De-Warp
    • Motion Estimation
    • ISP Pipeline
    • JPEG Encoding
    • H.264 and H.265 Encoding

    The How and What of DepthAI:

    So to allow the use of these, DepthAI is engineered to allow direct MIPI connection:

    And to allow this power to be integrate-able into actual products (which may require differing stereo baselines, differing interfaces, geometries, etc.) and to make hardware integration much easier (removing the need to integrate a very-fine-pitch multi-hundred-ball BGA into your design) we made a System on Module to hold the Myriad X:

    This module enables a simple and standard tolerance PCB, instead of the high-layer-count and high-density-integration (including laser vias and stacked vias) required to integrate the Myriad X directly. And it also provides all the power supply rails, power sequencing, clock generation, and camera-support hardware on-module.

    This allows you to integrate the power of the Myriad X into your design with a standard 4-layer (or even 2-layer, gasp!) PCB. And then leverage our reference hardware and software designs to get up/going super-fast (or just buy our boards directly, if they happen to fit the bill for what you need). Which all results in getting up/running fast: DepthAI Models

    So what are the reference designs (the carrier boards) which will be released should the campaign be funded?

    DepthAI | Raspberry Pi Compute Module

    This design allows you to have the Myriad X, all the cameras, and the Raspberry Pi on one board. So that you don't have to worry about cabling things together, boards strewn about your desk, etc. when prototyping and writing code.

    DepthAI | Raspberry Pi HAT with FFC Camera Interfaces

     DepthAI Models DepthAI Models

    This design allows you to leverage...

    Read more »

  • Compute Module (BW1097) Second Revision

    Brandon10/16/2019 at 04:37 0 comments

    Hi Everyone,

    So we received the second revision of DepthAI for Raspberry Pi Compute Module, and everything works as hoped (and it looks a lot cooler because we made the design more compact and ordered it in black).  Pictures to follow.  :-)

    We're continuing to integrate the depth portions we've written (here) with the direct-from-image-sensor neural inference (object detection, in this case) above.

    Cheers!

    The Luxonis Team

View all 46 project logs

Enjoy this project?

Share

Discussions

Andrey V wrote 08/30/2019 at 17:53 point

I think it's too hard to get it work IRL. If you will do it, your LVL=80)

  Are you sure? yes | no

Brandon wrote 08/30/2019 at 17:59 point

Heh.  Thanks.  It's definitely a hard project - but we're hoping it will provide a lot of value!

  Are you sure? yes | no

Andrey V wrote 08/30/2019 at 18:02 point

Good luck!!!

  Are you sure? yes | no

Alan wrote 08/04/2019 at 00:40 point

Does your SOM break out the PCIe lanes on the movidius? I was looking at the UP Ai core but they convert PCIe to usb on board.

  Are you sure? yes | no

Brandon wrote 08/04/2019 at 01:03 point

Great question, thanks.

It does not.  The SoM uses USB3 (or 2) to connect to the host.  Less power, fewer parts, and also works with hosts that don’t have PCIE or don’t want to tie up a PCIE slot for this.

On the DepthAI for Raspberry Pi version there’s a USB hub chip on the carrier board, which is what allows the pi compute module to talk to the myriad x and also the outside world.

And yes the PCIE boards do convert from PCIE to USB on-board.

If you don’t mind me asking, why do you ask?  Do you need direct PCIE connection?

Thoughts?

Thanks again,

Brandon

  Are you sure? yes | no

Alan wrote 08/05/2019 at 05:47 point

I can't think of any reason right away, but it would be great to have as much of the io exposed as possible.  I think at least the SPI and ethernet should be exposed because those would both be useful for someone who wanted to use the SOM as a standalone device.

P.s. what did you guys have to do in order to aquire the chips? I can't seem to find anywhere to order them online

  Are you sure? yes | no

Brandon wrote 08/06/2019 at 22:35 point

Hey Alan,

So for some reason I can't reply to your comment, so I'm replying here instead.  So we don't know how to make Ethernet work yet from a firmware standpoint (and don't have a clear path to figuring it out) so we left it off of this SoM.  

That said we are making a SoM w/ an i.MX 8M on w/ the Myriad X, so that would provide Ethernet and a whole slew of interfaces like USB3, WiFi, etc.

Thoughts?

Thanks,

Brandon

  Are you sure? yes | no

psykhon wrote 03/07/2019 at 12:19 point

Hi Brandon, awesome project! 

How hard was to get te myriad x chips? Can you share some info on how do you do it? price?

  Are you sure? yes | no

Tegwyn☠Twmffat wrote 02/01/2019 at 14:53 point

I checked out that link above - I wonder how their larger kit compares to the Jetson TX2 in terms of performance?

I realise performance is not everything and the Intel model zoo is pretty useful. The Nvidia software seems to be a bit behind in that they only have blvc_Googlenet as 'out of the box' solution for detection.

What do you think your price point will be for a single myriad X carrier board, I'm presuming about $100 ?

  Are you sure? yes | no

Brandon wrote 02/01/2019 at 15:02 point

Great question!  So we've actually done a decent amount of stuff on the Tx2 as well.  The Myriad X, in terms of straight neural inference performance (e.g. object detection, semantic segmentation, etc.) is about the same as the Tx2.  The Myriad X neural engine is 1 TOPS, and the Tx2 peaks in ideal conditions at 2 TOPS, but from below, it seems like in most conditions, it's effectively 1 TOPS:

https://devtalk.nvidia.com/default/topic/1024825/cuda-programming-and-performance/jetson-tx2-performance/

But!  If your application is depth vision + neural inference, the Myriad X is equivalent to about 2 or 3 Jetson Tx2, mainly because of the 16 SHAVE cores in the Myriad X, which together can do 6 cameras in 3 pairs of depth streams. 

The neural inference part of the Myriad X is only 1 TOPS of the total 4 TOPS the device an do.  The remaining tops are for image processing functions like depth vision.

So this board won't really even tax the Myriad X, as there will just be one depth stream.  That said, we can use the extra Myriad X 'head room' to run fancier/more-processing-intensive depth calculation on these just 2 cameras - to produce a better set of depth information.

  Are you sure? yes | no

Tegwyn☠Twmffat wrote 01/31/2019 at 22:58 point

Hello Brandon! Does the Myriad X chip get put on the carrier board or does it stay in the USB stick?

If it goes on the board, how many of them?

  Are you sure? yes | no

Brandon wrote 02/01/2019 at 12:37 point

The Myriad X would be directly on the carrier board.  We could make versions with multiple Myriad X, for sure.  Is that of interest?  

These guys did that for their PCIE version:

https://www.crowdsupply.com/up/ai-core-x

I have 2 of those on order, by the way.  They're useful as well, for sure - just a different application, and not applicable for the Pi community (which is what this board should serve).

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates