Close
0%
0%

Lepton 3.5 Thermal Imaging Camera

Documenting my experiments with the FLIR Lepton 3.5 thermal imaging camera.

Similar projects worth following
I bought a FLIR Lepton 3.5 mounted on the Pure Engineering breakout after using a friend's thermal imaging camera to analyze heat generation on a PCB I designed. The Lepton 3.5 can output absolute temperature data (Radiometric) at 160x120 pixel resolution. I decided a thermal imaging camera would be a good tool to have and decided to build my own. I started with a Teensy 3.2 based test platform, turned to a Beaglebone Black with a short stop in Raspberry Pi land, and finally developed a set of thermal imaging cameras based around the ESP32. I hope the documentation in this project is helpful to others who might also want to play with these amazing devices.

My original long-term goal was to create a capable thermal imaging camera using the Beaglebone Black and a 7" LCD cape as the platform matching some of the features of high end commercial products.  However it soon became obvious that I'd need a simpler platform to learn how to use the Lepton module when I started reading the documentation and playing with the various demo codebases.  It is a very capable device with a moderately complex interface, both firmware and hardware.  Although the device has good default settings I found enabling some features wasn't well documented and the video SPI interface (VoSPI) challenging to implement due to its real-time constraints.


There are a lot of other great projects online to help get going with the FLIR sensors.  Pure Engineering is to be commended for making these devices available to makers and provides a wealth of code examples, many designed to work with the previous Lepton models.  Max Ritter's DIY Thermocam is probably the most mature and well known.  He has done a great job and I pored over his code.  Damien Walsh's Leptonic is also really well done and works with the Lepton 3.5 as well.  Both Max and Damien were very gracious when I sent them various questions.

I decided to follow Max's lead and build a test platform using a Teensy 3.5 that I had (selected for the multiple SPI interfaces and copious RAM).  Unfortunately after soldering the Teensy to a Sparkfun breakout board I stressed the processor BGA package and made the board unreliable.  So I replaced it with a Teensy 3.2 hoping it would have enough resources to successfully interface to the Lepton.  It does, barely, and the next project log describes the test platform hardware.

teensy_schematic.pdf

Test platform schematic

Adobe Portable Document Format - 26.67 kB - 07/10/2018 at 19:11

Preview
Download

View all 18 components

  • tCam/tCam-Mini serial programmer utility

    Dan Julio5 days ago 0 comments

    I wrote a simple utility program that runs on 64-bit Linux x86, OS X or Windows to make it easy to load the most recent firmware (downloaded from my website) for either tCam (gCore hardware) or tCam-Mini/tCam-POE.  I did this primarily to make it easy to program a gCore for those people who are building tCams but it can also be used to load the most current tCamMini firmware as well.

    It simply requires connecting the camera's USB serial port to the computer and programming.

    It can be downloaded from my website.  Look for the tCam Family Serial updater section.

    The application itself is pretty simple xojo.  It's able to program the ESP32 on the different platforms because it includes compiled versions (for each platform) of the Espressif esptool.py python program.  I just use xojo's shell object to run the binary version with the selected serial port and paths to the downloaded firmware binary files.

    I was led to this idea when I found that the Espressif ESP32 Arduino package included compiled versions for Windows and Mac OS X.  I dug around their github repositories until I found a script that uses pyinstaller to compile the python esptool.py.  I only had to create a binary for Linux x86 which I did with the following command after installing pyserial and pyinstaller.

    pyinstaller -F --add-data="esptool/targets/stub_flasher/*.json:esptool/targets/stub_flasher/" esptool.py

    This left a built executable called esptool in a dist directory.  Very cool.

  • tCam-POE design pushed to github

    Dan Julio01/22/2023 at 19:46 0 comments

    I finally had a moment to add the tCam-POE hardware design to the tCam github repository.   tCam-POE is basically tCam-Mini but with the ability to connect via ethernet instead of Wifi and also supports IEEE802.3af POE power.  No word yet if Group Gets is planning to build this.  Let them know if you would like to see it as a product.  :-)

  • tCam Video

    Dan Julio12/13/2022 at 04:06 0 comments

  • tCam! Finally (and a github change)

    Dan Julio11/06/2022 at 20:18 0 comments

    gCore is going to be available through a campaign at Group Gets.  This means tCam can also finally become a reality for anyone who would like to make one.  Finally.  It has felt, at times, that I'd never get to this point.

    There's a dedicated gCore github repository.  

    And I moved all the ESP32 thermal imaging stuff (tCam-Mini, apps, etc) to its own github repository so that people who are interested only in that don't have to download everything else I did with the Lepton.  The tCam firmware has been uploaded there as well as instructions for use and the enclosure designs.  The firmware binaries which must be loaded onto gCore and the enclosure designs can also be directly downloaded from my website.

    I realize that gCore might seem a bit pricey at the moment.  Neither I or Group Gets is making much money on this campaign.  It's really a factor of the parts cost (higher than some dev boards because gCore has a lot of high end features, and those take extra silicon), the manufacturing cost as Group Gets is building the boards with a USA contract manufacturer and the initial low volume of only 100 boards.  Even so I think it allows building a full feature, open-source thermal imaging camera for a reasonable price.  Over time I am working to bring this price down.  That will be discussed on the gCore project page.

  • Big Software Release

    Dan Julio08/08/2022 at 21:00 0 comments

    Hardware designs for gCore and tCam-POE have gone to Group Gets for quoting and buying.  I've noticed Mouser has a lot more parts recently so maybe the great partageddon is finally showing signs of easing.

    I also just pushed a bunch of new code up to github and my website.

    tCam-Mini Firmware Revision 3.0

    Big changes so this code base can also support tCam-POE.  I also added a mDNS (bonjour) responder so that the IP addresses of cameras on a network can be found by the various applications.  Rounding this release out were a few bug fixes plus I changed the internal flash access to use 80 MHz QIO accesses in preparation for some future firmware enhancements.

    Sadly the day after I made the release I found another minor bug.   A camera that has had its WiFi reconfigured to use STA mode with a static IP address may not correctly restart the WiFi stack until the next reboot.  Easy fix that will be rolled out in a bit but odd that I didn't see this before.

    A very kind soul got the previous firmware to compile on Espressif IDF 4.4.1 (instead of my ancient 4.0.2) and sent a pull request so I'll be trying to integrate his work for the next release too.

    Desktop Application 3.0.0

    A whole bunch of changes so I'm just copying the release notes here.

    1. New IsoTherm palette
    2. Added ability to change palettes by clicking the top or bottom of the color bar
    3. Added support to identify new Ethernet camera model
    4. Added Camera Info window (opened by new Camera->Camera Info... menu item)
    5. Application no longer displays firmware version of connected camera in main window
    6. Added new Discover function for bonjour discovery of cameras from Preferences Window
    7. Wifi/Network Settings tab understands Ethernet camera model and disables irrelevant controls
    8. Added Help window accessed through Application->Documentation menu
    9. Added ability to check for updated application automatically at startup or from Application Menu
    10. Added ability to check for new tcam/tCamMini firmware versions from Camera Menu based on connected camera type
    11. Enabled close button on About window
    12. Main Window toolbar connect button toggles between "Connect" and "Disconnect" captions
    13. Improved error message when application can't connect to camera
    14. Internal changes with window management in preparation for multi-camera sessions
    15. Signed Mac version with Apple developer ID

    Here's a screen shot of the new mDNS camera discovery in operation (accessed from the Preferences Window where  you set the camera address).

    Perhaps the neatest feature of the new version is the ability of the application to tell you if there is an updated version of itself or an updated version of the firmware for the attached camera and then download a zip file with the new code.

    iOS App

    The iOS app got the new IsoTherm palette as well as camera discovery.  The IsoTherm palette is great for quickly finding both the hot and cold spots in an image.

    Python Driver

    Last, but not least,  my friend bitreaper and I have been working on the Python driver to allow it to support the tCam-Mini Hardware Interface on a Raspberry Pi.  One of the things I've noticed over the years (and saw for myself) is that user space programs on the Pi have a hard time keeping up the the Lepton VoSPI pipeline.  Lots of users complaining in the forums about that. tCam-Mini with the python driver is a great solution to that problem.  In addition to communicating over a network socket the Python driver can now use the Pi's serial and SPI ports to communicate directly with a tCam-Mini.

    My very first tCam-Mini PCB got new use with wires soldered directly to the ESP32 module for testing.  Here it is on a Raspberry Pi 4 running the streamtest_hw.py demo and a full speed streaming image.

    The driver makes it easy to process data from the camera.  After setup the following two routines handle getting the radiometric data from the stream and converting it to an image to display using Tk.

    def convert(img):
    
        dimg = base64.b64decode(img[...
    Read more »

  • Introducing tCam-POE

    Dan Julio07/28/2022 at 22:01 0 comments

    I've been working on a new tCam model that helps deal with the limitations of WiFi in some situations, for example monitoring temperature inside metal cabinets or other industrial environments.  The tCam-POE design is based on the tCam-Mini design but also includes support for wired ethernet through the inclusion of an ethernet PHY chip and magnetics.  In fact it and tCam-Mini will run the same firmware starting with the upcoming version 3.0 release.  tCam-POE can connect using either 10/100 ethernet or WiFi (switch selectable at power up) and it can be powered via a USB-C 5V power source or, with the additional of a commonly available IEEE 802.3af DC-DC converter module, via power-over-ethernet.

    Shown here are my two prototypes sporting Winchen and Silvertel POE DC-DC converters.

    tCam-POE has the same command set as tCam-Mini with one exception.  The set_wifi command doesn't need to include WiFi-specific arguments such as SSID and Password and tCam-POE cannot be configured to serve DHCP addresses over ethernet (it still can act as an AP when running with the WiFi interface).  It 's ethernet interface either gets a DHCP served address from an external router or it is set with a static IPv4 address.  It also doesn't have the hardware interface that tCam-Mini Rev 4 PCB has.

    Porting the code to support it was pretty easy as Espressif has built-in support for an ethernet interface and after initialization the same network stack can be used for either WiFi or ethernet.

    Small changes are being made to the other apps so they can identify the new model number sent by tCam-POE.

    All code will be put up in the next couple of weeks.

    The plan is for Group Gets to build and stock these units.  Hopefully in the next 2-3 months - pending parts availability (I'm getting better about specifying parts with multiple sources...)

  • An [Android] tease

    Dan Julio07/21/2022 at 17:59 0 comments

    While I wait for Xojo to get their act together wrt Android support, a friend decided he could use my project as an excuse to refresh his Android development skills.  His work is paying off and he sent me an alpha-level APK to play with today.  For everyone who has patiently been waiting for an Android App that can talk to the tCam family of cameras, I present this tease.

    If all goes well there will be a beta APK file for people to side load in a few weeks.  My friend's longer-term goal is to get the app into the Play store.

  • Whither tCam

    Dan Julio06/02/2022 at 18:08 0 comments

    It's crazy sometimes how fast time seems to fly by.  I have been working on tCam for over two years now in fits and starts.  I can see how someone reading these project log entries might suppose it's a dead project.   It isn't but it also isn't where I wish it were.

    The good news is that the code is essentially done and the camera works.  Shown above with a streaming display also shown on the iOS mobile app.  I am currently working toward FW 3.0 for tCam-Mini and will include its new functionality in tCam as well and that will be tCam FW 1.0 running on gCore.

    In addition I'm working with Group Gets to bring gCore to market as a general purpose ESP32 development board for building GUI-based battery powered gadgets.  The gCore design is done and I even have an Arduino library and a bunch of demos written for it.  I think it's going to be very cool.

    The frustrating news is the parts shortage is making it hard to build gCore.  It may end up with a slight redesign to accommodate available parts. 

    I am considering pushing the code to github even though it won't be super useful without gCore (or some modification).

    Here are a few more pics showing how tCam is easily built using a Revision 4 tCam-Mini and gCore in a laser-cut enclosure.  The jumper on tCam-Mini configures it to use the hardware serial interface to communicate with gCore.  Hopefully there will also be a 3D printed enclosure design.

  • tCam-Mini (rev 4 PCB) available

    Dan Julio04/29/2022 at 16:27 0 comments

    Despite the ongoing parts armageddon, the gang at Group Gets managed to find enough parts to build the new Revision 4 tCam-Mini (Rev 3 won't see the light of day outside of 3 prototypes).

    They even surprised me by creating a version that supports an external antenna - great for applications like flying the camera on a drone.

    Photos courtesy of Group Gets.  The new cameras can be found here and here.

    This version is a bit modernized with a USB-C connector and has connections for the new direct hardware interface supported in firmware versions 2.0 and beyond.  Designed so I could use tCam-Mini as the imager for tCam, the hardware interface allows direct connection of tCam-Mini to another micro-controller or SBC.  It allows tCam-Mini to handle all Lepton VoSPI related issues and provides the same high-level interface as WiFi.  It consists of a serial port running at 230,400 baud and a slave SPI port.  The same json commands and responses are transmitted over the serial port with one exception.  Instead of sending an image over the serial port (which would be too slow) the camera sends a new "image_ready" json packet that also includes the length of the image.  Then the controller can use a SPI master to read the image from the slave SPI port on tCam-Mini.  Full details can be found on the github readme.  The 3.3v logic-level hardware interface is activated when the MODE input is detected low at boot (otherwise the WiFi interface is activated).

    The github repository has also been updated with the new hardware design files.

  • New 3D printed tCam-Mini enclosure

    Dan Julio03/02/2022 at 02:41 0 comments

    I finally had some time, after the holidays, to build a Prusa i3 MK3 3D printer and I've been having fun using it.  It's a fine machine.  I also took a little time to design an enclosure for tCam-Mini.   The basic enclosure is comprised of two pieces, a base and a bezel, held together with 4 4-40x5/8" metal screws with the tCam-Mini board captured between.  There are two different designs for the base.  One for the existing Rev 2 PCB and one for the upcoming Rev 4 design.

    There is also a three piece GoPro™ compatible mount that clips onto the enclosure.  It's light and designed to be the thing that breaks to support tCam-Mini on a drone.

    The design is in OpenSCAD and both the design and STL files ready for slicing may be found on my website or in the github repository.

View all 49 project logs

Enjoy this project?

Share

Discussions

Nathan Sinclair wrote 4 days ago point

Hello Dan. having limited knowledge about sockets, I have a general query.  What is the reason for using ZeroMQ? From the little research I've done so far the message size for even 1 frame would be way too high... am I missing something? 

  Are you sure? yes | no

sullivan wrote 08/09/2021 at 18:53 point

Hi Dan. Love the project. I have been playing with a Lepton 3.5 on the breakout board 2 and Raspberry Pi 4B. The thermal images I am getting are in the range of 8300, not the radiometric, T-linear mode I am expecting. Do you get images in the 30000 range where the formula: T(degC) = pixval/100 - 273.15? If you are also in the range of 8300, how are you converting to temperature? Thanks in advance for the advice.

  Are you sure? yes | no

Dan Julio wrote 10/06/2021 at 03:15 point

Hey Sullivan, I'm so sorry for this tardy reply.  For some reason I no longer seem to get notifications when someone posts here.  Are you sure you're switching the camera into radiometric mode with TLinear?  If so then what gain level is your lepton running at?  In High gain mode the resolution is 0.01° which means the data is in the 30000 range (e.g. 25°C -> count of 29815) but in Low gain mode the resolution is 0.1°C (25°C -> 2981).  In my cameras I set Radiometry, Radiometry TLinear, AGC Calcs all true.

  Are you sure? yes | no

leoshmu wrote 01/12/2021 at 03:23 point

Thank you for this excellent project. I am curious about pushing the limits of frame rate. Are you able to achieve a full 9Hz frame rate on the thermal video stream, or do you find that the frame rate is limited to less than 9Hz in practice. Could you imagine getting 2 leptons integrated together and offsetting the thermal stream so that they are off by 55ms, allowing for a combined 18hz signal? Perhaps something with a master clock circuit and then controlling each lepton separately based on the clock timing? Also I'm curious, why do you prefer the pure thermal board to the lepton v2 breakout board? 

  Are you sure? yes | no

Dan Julio wrote 02/04/2021 at 03:26 point

Hey Leo, thanks for the kind words.  I have been able to achieve the full 9Hz frame rate using the Beaglebone's PRU (in AGC mode) and with the ESP32 (both radiometric and AGC modes).  With the ESP32 I more-or-less dedicate one of the two CPUs to getting data out of the Lepton (and I use the VSYNC signal).  I will shortly be posting the code for tCam-Mini which is capable of reading the full frame rate (but not quite capable of sending that frame-rate over the WiFi).

Regarding your interesting idea of using two devices:  I'm not sure how to synchronize them.  I think there is a lot of internal processing that ends up affecting when the device is capable of sending a video packet - and things like a FFC also impact its ability to output a valid frame.  I suspect - but have no proof - that it is capable of a higher frame rate but held back in firmware (the sensor response time seems to be around 50 mSec).  I'm not up to it but I've wondered if someone could hack it.

I like the pure thermal board because it's smaller - I don't think there are any performance related differences.  Now, with tCam-Mini, I'm moving to directly mounting the sensor on my own boards.  

  Are you sure? yes | no

Ailsamly wrote 08/18/2020 at 10:30 point

Hi Dan,

I also plan to build a small thermal camera with a  suitable thermal sensor module, I want to connect it to a mobile platform and do building inspection (like water damage and air leakage and other thermal anomalies) .  The thermal sensor will be 5-10cm in front of the wall. I want to follow Max Ritter's project DIY-Thermocam. But I am not sure whether the resolution of Flir lepton can fulfull the requirements of mine. Do you have some ideas? Thank you!

Best,

Leyuan

  Are you sure? yes | no

Dan Julio wrote 12/26/2020 at 19:11 point

Hello Leyuan, I apologize for this tardy reply.  For some reason I never saw your post.  I think the Lepton should work from a distance of 5-10 cm.  However I can't answer your question about the resolution as I'm not sure what the characteristics of the things you are looking for are.  I can only suggest seeing if you could borrow a camera from someone and do some experimentation before committing to your own build.  Good luck though!  Sounds like a cool project.

  Are you sure? yes | no

Richard wrote 05/30/2020 at 16:15 point

Hi Dan,

According to Flir's datasheets, the Lepton uses 3.0V for IO functions, but the Teensy's on board Vreg runs at 3.3V. Did you do any level translation on the VOSPI and I2C interfaces or modify the Teensy to run at 3.0V, or just put 3.3V on the buses? I'm about to start a Lepton project of my own and want to be sure I don't kill my expensive sensor!

Thanks!

  Are you sure? yes | no

Dan Julio wrote 05/30/2020 at 16:49 point

Hey Richard,

You're right, they specify a 2.8-3.1 volt VDDIO.  But they also specify that the maximum on an IO pin is the lesser of VDDIO + 0.6 volts or 4.8 volts (table 18 - Absolute Maximum Ratings).  The Pure Thermal breakout boards have a 2.8 volt regulator feeding VDDIO so the maximum input voltage should be 3.4 volts.

To be honest I didn't pay attention to this spec when I first started.  I saw that the Pure Thermal boards were used with many different 3.3V SBCs so I jumped right in.  It wasn't until later I saw the specs you are referring to.

Although it doesn't seem like good engineering practice, I have run several Lepton 3.5 modules for long periods of time and this does not seem to change their functionality.  One could design their own breakout board for the Lepton and provide 3.1 volts to VDDIO which would lessen the over-voltage condition even more or provide resistor dividers on the SCLK and CSN signals (and pull SDA/SCL to the VDDIO rail).  As I work on what I hope is my final thermal imaging camera I am contemplating a full custom PCB (eliminating the breakout boards) and this is probably what I'll do, especially after your timely email :-)

  Are you sure? yes | no

Avi Cohen wrote 11/14/2019 at 11:45 point

Hi Dan, 

Is the code open to download?

  Are you sure? yes | no

Dan Julio wrote 11/14/2019 at 17:09 point

Hey Avi, 

yes, it's in the github repository.

https://github.com/danjulio/lepton

  Are you sure? yes | no

Sophi Kravitz wrote 07/10/2018 at 17:21 point

hi Dan! This is a cool project! This is happening pretty soon: https://hackaday.io/contest/159585-battery-powered-radiometric-lepton-contest

  Are you sure? yes | no

Dan Julio wrote 07/10/2018 at 18:09 point

Hi Sophi!  I saw that.  I'm not sure I'll have anything for the Lepton 2.5 but I will contribute a port of FLIR's LeptonSDKEmb32OEM library for the Teensy which might be applicable to the ESP32.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates