Luxonis DepthAI

Spatial AI Meets Embedded Systems

Public Chat
Similar projects worth following
Machine learning (ML) based computer vision (CV) is incredibly powerful... but when you go to use it to interact with physical world it can be incredibly frustrating and limiting.

This is because so far the whole ML-based computer vision has been stuck in 2D, particularly when it comes to Edge or Embedded AI.DepthAI brings Spatial AI - unleashing the power of machine learning into 3 dimensions.

Real-time 3D results of what object are, and where it is in x, y, and z relative to the camera, on your embedded platform (SPI interface), your Raspberry Pi (USB), or Linux/Mac/Windows computer. With convenient (and free!) Google Colab notebooks for training on your objects of interest.

With Open Source hardware and software integration and modification for your prototype and product is painless and low risk:

We have launched our KickStarter Campaign!

Today, our team is excited to release to you the OpenCV AI Kit, OAK, a modular, open-source ecosystem composed of MIT-licensed hardware, software, and AI training - that allows you to embed Spatial AI and CV super-powers into your product.

And best of all, you can buy this complete solution today and integrate it into your product tomorrow.

Back our campaign today!

The Why

  • There’s an epidemic in the US of injuries and deaths of people who ride bikes
  • Majority of cases are distracted driving caused by smart phones (social media, texting, e-mailing, etc.)
  • We set out to try to make people safer on bicycles in the US
    • We’re technologists
    • Focused on AI/ML/Embedded
    • So we’re seeing if we can make a technology solution

Commute Guardian

(If you'd like to read more about CommuteGuardian, see here)

DepthAI Platform

  • In prototyping the Commute Guardian, we realized how powerful the combination of Depth and AI is.
  • And we realized that no such embedded platform existed
  • So we built it.  And we're releasing it to the world through a Crowd Supply campaign, here

We want this power to be easily embeddable into products (including our own) in a variety of form-factors (yours and ours).  So we made a System on Module which exposes all the key interfaces through an easy-to-integrate 100-pin connector.  

Unlike existing USB or PCIE Myriad X modules, our DepthAI module exposes 3x MIPI camera connections (1x 4-lane, 2x 2-lane) which allows the Myriad X to received data directly from the camera modules - unburdening the host

The direct MIPI connections to the Myriad X removes the video data path from the host entirely.  And actually this means the Myriad X can operate without a host entirely.  Or it can operate with a host, leaving the host CPU completely unburdened with all the vision and AI work being done entirely on the DepthAI module/Myriad X.

This results in huge efficiency increases (and power reduction) while also reducing latency, increasing overall frame-rate, and allowing hardware blocks which were previously unusable to be leveraged.

Take real-time object detection on the Myriad X interface with the Raspberry Pi 3B+ as an example:

Because of the data path efficiencies of DepthAI vs. using an NCS2, the frame rate increases from 8 FPS to 25FPS.  

And most importantly, using this data path allow utilization of the following Myriad X hardware blocks which are unusable with previous solutions:

This means that DepthAI is a full visual perception module - including 3D perception - and no longer just a neural processor, enabling real-time object localization in physical space, like below, but at 25FPS instead of 3FPS:

And to allow you to use this power right away, and with your favorite OS/platform, we made 3x editions of DepthAI, which serve as both reference designs for integrating the DepthAI module into your own custom hardware and also as ready-to-use platforms that can be used as-is to solve your computer vision problems, as-is.
  1. Raspberry Pi Compute Module Edition - with 3x integrated cameras
  2. Raspberry Pi HAT Edition - with 3x modular cameras
  3. USB3 Edition - compatible with Yocto, Debian, Ubuntu, Mac OS X and Windows 10

All of the above reference designs will be released should our CrowdSupply campaign be successfully funded.  So if you'd like to leverage these designs for your designs, or if you'd like to use this hardware directly, please support our CrowdSupply campaign:

Development Steps

The above is the result of a lot of background work to get familiar with the Myriad X, and architect and iterate the System on Module definition, form-factor, interfaces, and manufacturability.  Below are some of the steps involved in that process.

The first thing...

Read more »

  • 1 × Intel Movidius Myriad X Vision/AI Processor
  • 1 × CM3B+ Raspberry Pi Compute Module 3B+
  • 2 × OV9282 Global Shutter camera modules optimized for disparity depth
  • 1 × IMX378 Nice high-resolution 12MP camera module that supports 12MP stills

  • Bluebox Co-Pilot AI Just Launched Based on DepthAI!

    Brandon07/09/2021 at 17:49 0 comments
  • DepthAI API 2.6.0 Release

    Brandon07/06/2021 at 02:28 0 comments

    • Added EdgeDetector node, using 3x3 HW sobel filter
    • Added support for SpatialCalculator on RGB-depth aligned frames; Note: RGB ISP and depth frame must have the same output resolution (e.g. 1280x720), maximum frame width of 1920 pixels is supported
    • Added bilateral filter on depth frame
    • Added median filter after stereo LR-check
    • Added runtime configuration for stereo node: confidence threshold, LR-check threshold, bilateral sigma, median filter
    • Added calibration mesh support for fisheye lenses
    • Bumped c++ standard to c++14




  • POE Models, Fun Edge Filter Hardware Block

    Brandon06/23/2021 at 15:48 0 comments

    1. We're about to release our POE variants of DepthAI.  
    2. We have the first version of the Edge Filter hardware block running.

    POE Models working:

    Edge HW Filter:

    Cool Community Projects:





  • The Gen2 Version of Our API Is Out

    Brandon03/02/2021 at 16:47 0 comments

    Hi Everyone,

    We just released Gen2.

    First Gen2 release (2.0.0 and for python bindings) is now online:
    - Core:
    - Python:

    Core now supports find_package(depthai) exposing targets depthai::core and depthai::opencv (opencv support)
    Python has docstring built into wheels, so help should display reasonable information for the function.

    Also documentation for gen2 (API reference) includes both C++ and Python fully documented with all available functions, classes, ...
    The API reference is just an overview of everything - we are working on more written documentation for messages, nodes, ...

    We also have our IP67 depthai POE prototypes in:

    Design files are here:

    And check out this sweet auto-following robot using DepthAI from Scott Horton:

    Notes from him:

    A bit of success using the OAK-D + Depthai ROS-2 sample with the ROS-2 Nav stack. The Nav stack is running the dynamic object following mode ('follow-point' behavior tree). Running on an RPI4 (the Intel camera and jetson are just along for the ride).

    Join our Discord to see even more in action.  It's really fun for the whole depthai team to see the things that are being actively made now.  

    We also have a 20MP 1" diagonal huge-sensor support coming:

    And wide angle support is now available via ArduCam as well.  They made these specifically for DepthAI:

    There's a TON more that we're not covering here (out of time), but join our Discord to learn more and discuss:


    Brandon and the depthai team

  • A Whole Bunch of New DepthAI Capabilities

    Brandon12/18/2020 at 04:34 0 comments

    Hi DepthAI Backers!

    Thanks again for all the continued support and interest in the platform.

    So we've been hard at work adding a TON of DepthAI functionalities.  You can track a lot of the progress in the following Github projects:

     - Gen1 Feature-Complete:
     - Gen2 December-Delivery:
     - Gen2 2021 Efforts: (Some are even already in progress)

    As you can see, there are a TON of features we have released since the last update.  Let's highlight a few below:

    RGB-Depth Alignment

    We have the calibration stage working now.  And future DepthAI builds (after this writing) are actually having RGB-right calibration performed.  An example with semantic segmentation is shown below:

    The `right` grayscale camera is shown on the right and the RGB is shown on the left.  You can see the cameras are slightly different aspect ratios and fields of view, but the semantic segmentation is still properly applied.  More details on this, and to track progress, see our Github issue on this feature here:

    Subpixel Capability

    DepthAI now supports subpixel.  To try it out yourself, use the example [here](  And see below for my quickly using this at my desk:

    Host Side Depth Capability

    We also now allow performing depth estimation from images sent from the host.  This is very convenient for test/validation - as stored images can be used.  And along with this, we now support outputting the rectified-left and rectified-right, so they can be stored and later used with DepthAI's depth engine in various CV pipelines.  

    See [here]( on how to do this with your DepthAI model.  And see some examples below from the MiddleBury stereo dataset:

    For the bad looking areas, these are caused by the objects being too close to the camera for the given baseline, exceeding the 96 pixels max distance for disparity matching (StereoDepth engine constraint):

    These areas will be improved with `extended = True`, however Extended Disparity and Subpixel cannot operate both at the same time.

    RGB Focus, Exposure, and Sensitivity Control

    We also added the capability (and examples on how to use) manual focus, exposure, and sensitivity controls.  See [here]( for how to use these controls. Here is an example of increasing the exposure time:

    And here is setting it quite low:

    It's actually fairly remarkable how well the neural network still detects me as a person even when the image is this dark.

    Pure Embedded DepthAI

    We mentioned in our last update ([here](, we mentioned that we were making a pure-embedded DepthAI.  

    We made it.  Here's the initial concept:

    And here it is working!

    And here it is on a wrist to give a reference of its size:

    And [eProsima]( even got microROS running on this with DepthAI, exporting results over WiFi back to RViz:

    RPi Compute Module 4

    We're quite excited about this one.  We're fairly close to ordering it.  Some initial views in Altium below:

    There's a bunch more, but we'll leave you with our recent interview with Chris Gammel at the Amp Hour!


    Brandon & The Luxonis Team

  • Announcing OpenCV AI Kit (OAK)

    Brandon07/14/2020 at 13:48 0 comments

    Today, our team is excited to release to you the OpenCV AI Kit, OAK, a modular, open-source ecosystem composed of MIT-licensed hardware, software, and AI training - that allows you to embed Spatial AI and CV super-powers into your product.

    And best of all, you can buy this complete solution today and integrate it into your product tomorrow.

    Back our campaign today!

  • megaAI CrowdSupply Campaign Production Batch Complete

    Brandon07/09/2020 at 19:58 0 comments

    Our production run of the megaAI CrowdSupply campaign is complete and now shipping to us:

    We had 97% yield on the first round of testing and 99% yield after rework and retest of the 3% that had issues in the first testing.

  • Pure Embedded Variant of DepthAI

    Brandon07/02/2020 at 15:34 0 comments

    Hi DepthAI Backers and Fans,

    So we've proof-of-concepted an SPI-only interface for DepthAI and it's working well (proof of concept done with MSP430 and Raspberry Pi over SPI).

    So to make it easier for engineers to leverage this power (and also for us internally to develop it), we're making a complete hardware and software/AI reference design for the ESP32, with the primary interface between DepthAI and the ESP32 being SPI.

    The design will still have USB3C for DepthAI, which will allow you to see live high-bandwidth results/etc. on a computer while integrating/debugging communication to your ESP32 code (both running in parallel, which will be nice for debugging).  Similarly, the ESP32 will have an onboard UART-USB converter and micro-USB connector for programming/interfacing w/ the ESP32 for easy development/debug.

    For details and progress on the hardware effort see [here] and to check out the SPI support enhancement on DepthAI API see [here]

    In short here's the concept:

    And here's a first cut at the placement:

    And please let us know if you have any thoughts/comments/questions on this design!


    Brandon & The Luxonis Team

  • New Spatial AI Capabilities & Multi-Stage Inference

    Brandon06/17/2020 at 19:53 0 comments

    We have a super-interesting feature-set coming to DepthAI:

    • 3D feature localization (e.g. finding facial features) in physical space
    • Parallel-inference-based 3D object localization
    • Two-stage neural inference support

    And all of these are initially working (in this PR, [here](

    So to the details and how this works:

    We are actually implementing a feature that allows you to run neural inference on either or both of the grayscale cameras. 

    This sort of flow is ideal for finding the 3D location of small objects, shiny objects, or objects for which disparity depth might struggle to resolve the distance (z-dimension), which is used to get the 3D position (XYZ). So this now means DepthAI can be used two modalities:

    1. As it's used now: The disparity depth results within a region of the object detector are used to re-project xyz location of the center of object.
    2. Run the neural network in parallel on both left/right grayscale cameras, and the results are used to triangulate the location of features.

    An example where 2 is extremely useful is finding the xyz positions of facial landmarks, such as eyes, nose, and corners of the mouth. 

    Why is this useful for facial features like this?  For small features like this, the risk of disparity depth having a hole in the location goes up, and even worse, for faces with glasses, the reflection of the glasses may throw the disparity depth calculation off (and in fact it might 'properly' give the depth result for the reflected object).

    When running the neural network in parallel, none of these issues exist, as the network finds the eyes, nose, and mouth corners per image, and then the disparity in location of these in pixels from the right and left stream results gives the z-dimension (depth = 1/disparity), and then this is reprojected through the optics of the camera to get the full XYZ position of all of these features.  

    And as you can see below, it works fine even w/ my quite-reflective anti-glare glasses:



    Brandon and the Luxonis Team

  • Raspberry Pi HQ Camera Works With DepthAI!

    Brandon06/10/2020 at 04:07 0 comments

    Hello everyone!

    So we have exciting news!  Over the weekend we wrote a driver for the IMX477 used in the Raspberry Pi HQ Camera.

    So now you can use the awesome new Raspberry Pi HQ camera with DepthAI FFC (here).  Below are some videos of it working right after we wrote the driver this weekend.

    Notice that it even worked w/ an extra long FFC cable!  ^

    More details on how to use it are here.  And remember DepthAI is open source, so you can even make your own adapter (or other DepthAI boards) from our Github here.

    And you can buy the adapter here:


    Brandon & the Luxonis team

View all 64 project logs

Enjoy this project?



rand3289 wrote 06/05/2020 at 03:00 point

Awesome project!  However, 3D is not enough.  Use TIME - the fourth dimension.  I describe why and what that means in this short paper: 

  Are you sure? yes | no

Brandon wrote 06/05/2020 at 03:13 point

Hi @rand3289 ,


Yes, time is very important in a lot of perception, whether it's integrating LSTM w/ YOLO, or using time to better estimate depth.

On the depth part, see here:

Which builds off of work from Google, which took advantage of time in the training data, but not in the network itself (which is the above improvement):

On YOLO with an LSTM (so to take advantage of time), see here:

I think there's newer/better than that.  I know AlexeyAB and he shared that w/ me back in July 2019... so that's from a LONG time ago in computer vision/AI rates.

We haven't investigated any of these on the platform... there's so much core platform work to be done, but many of them are probably runnable on the platform with some work/optimization.




  Are you sure? yes | no

Andrey V wrote 08/30/2019 at 17:53 point

I think it's too hard to get it work IRL. If you will do it, your LVL=80)

  Are you sure? yes | no

Brandon wrote 08/30/2019 at 17:59 point

Heh.  Thanks.  It's definitely a hard project - but we're hoping it will provide a lot of value!

  Are you sure? yes | no

Andrey V wrote 08/30/2019 at 18:02 point

Good luck!!!

  Are you sure? yes | no

Brandon wrote 06/05/2020 at 03:05 point

Hi @Andrey V - so you can buy it IRL now:  And it's open source with a whole ton of useful stuff.  To pick a random repo, checkout some of our recent experiments with the platform here:

  Are you sure? yes | no

Brandon wrote 10/05/2020 at 16:02 point

Hi @Andrey V ,

So as an update, you can also now buy this as the OpenCV AI Kit:
It comes with neat aluminum enclosures with Gorilla-Glass Anti-Reflective Coatings.  



  Are you sure? yes | no

Alan wrote 08/04/2019 at 00:40 point

Does your SOM break out the PCIe lanes on the movidius? I was looking at the UP Ai core but they convert PCIe to usb on board.

  Are you sure? yes | no

Brandon wrote 08/04/2019 at 01:03 point

Great question, thanks.

It does not.  The SoM uses USB3 (or 2) to connect to the host.  Less power, fewer parts, and also works with hosts that don’t have PCIE or don’t want to tie up a PCIE slot for this.

On the DepthAI for Raspberry Pi version there’s a USB hub chip on the carrier board, which is what allows the pi compute module to talk to the myriad x and also the outside world.

And yes the PCIE boards do convert from PCIE to USB on-board.

If you don’t mind me asking, why do you ask?  Do you need direct PCIE connection?


Thanks again,


  Are you sure? yes | no

Alan wrote 08/05/2019 at 05:47 point

I can't think of any reason right away, but it would be great to have as much of the io exposed as possible.  I think at least the SPI and ethernet should be exposed because those would both be useful for someone who wanted to use the SOM as a standalone device.

P.s. what did you guys have to do in order to aquire the chips? I can't seem to find anywhere to order them online

  Are you sure? yes | no

Brandon wrote 08/06/2019 at 22:35 point

Hey Alan,

So for some reason I can't reply to your comment, so I'm replying here instead.  So we don't know how to make Ethernet work yet from a firmware standpoint (and don't have a clear path to figuring it out) so we left it off of this SoM.  

That said we are making a SoM w/ an i.MX 8M on w/ the Myriad X, so that would provide Ethernet and a whole slew of interfaces like USB3, WiFi, etc.




  Are you sure? yes | no

psykhon wrote 03/07/2019 at 12:19 point

Hi Brandon, awesome project! 

How hard was to get te myriad x chips? Can you share some info on how do you do it? price?

  Are you sure? yes | no

Tegwyn☠Twmffat wrote 02/01/2019 at 14:53 point

I checked out that link above - I wonder how their larger kit compares to the Jetson TX2 in terms of performance?

I realise performance is not everything and the Intel model zoo is pretty useful. The Nvidia software seems to be a bit behind in that they only have blvc_Googlenet as 'out of the box' solution for detection.

What do you think your price point will be for a single myriad X carrier board, I'm presuming about $100 ?

  Are you sure? yes | no

Brandon wrote 02/01/2019 at 15:02 point

Great question!  So we've actually done a decent amount of stuff on the Tx2 as well.  The Myriad X, in terms of straight neural inference performance (e.g. object detection, semantic segmentation, etc.) is about the same as the Tx2.  The Myriad X neural engine is 1 TOPS, and the Tx2 peaks in ideal conditions at 2 TOPS, but from below, it seems like in most conditions, it's effectively 1 TOPS:

But!  If your application is depth vision + neural inference, the Myriad X is equivalent to about 2 or 3 Jetson Tx2, mainly because of the 16 SHAVE cores in the Myriad X, which together can do 6 cameras in 3 pairs of depth streams. 

The neural inference part of the Myriad X is only 1 TOPS of the total 4 TOPS the device an do.  The remaining tops are for image processing functions like depth vision.

So this board won't really even tax the Myriad X, as there will just be one depth stream.  That said, we can use the extra Myriad X 'head room' to run fancier/more-processing-intensive depth calculation on these just 2 cameras - to produce a better set of depth information.

  Are you sure? yes | no

Tegwyn☠Twmffat wrote 01/31/2019 at 22:58 point

Hello Brandon! Does the Myriad X chip get put on the carrier board or does it stay in the USB stick?

If it goes on the board, how many of them?

  Are you sure? yes | no

Brandon wrote 02/01/2019 at 12:37 point

The Myriad X would be directly on the carrier board.  We could make versions with multiple Myriad X, for sure.  Is that of interest?  

These guys did that for their PCIE version:

I have 2 of those on order, by the way.  They're useful as well, for sure - just a different application, and not applicable for the Pi community (which is what this board should serve).

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates