Close
0%
0%

Lepton 3.5 Thermal Imaging Camera

Documenting my experiments with the FLIR Lepton 3.5 thermal imaging camera.

Similar projects worth following
I bought a FLIR Lepton 3.5 mounted on the Pure Engineering breakout after using a friend's thermal imaging camera to analyze heat generation on a PCB I designed. The Lepton 3.5 can output absolute temperature data (Radiometric) at 160x120 pixel resolution. I decided a thermal imaging camera would be a good tool to have and decided to build my own. I have started with a Teensy 3.2 based test platform but eventually want to turn a Beaglebone Black with 7" LCD cape into a full-featured, networked, camera using the PRUs to handle the real-time video SPI feed from the Lepton. I hope the documentation in this project is helpful to others who might also want to play with these amazing devices.

The long-term goal is to create a capable thermal imaging camera using the Beaglebone Black and a 7" LCD cape as the platform matching some of the features of high end commercial products.  However it soon became obvious that I'd need a simpler platform to learn how to use the Lepton module when I started reading the documentation and playing with the various demo codebases.  It is a very capable device with a moderately complex interface, both firmware and hardware.  Although the device has good default settings I found enabling some features wasn't well documented and the video SPI interface (VoSPI) challenging to implement due to its real-time constraints.


There are a lot of other great projects online to help get going with the FLIR sensors.  Pure Engineering is to be commended for making these devices available to makers and provides a wealth of code examples, many designed to work with the previous Lepton models.  Max Ritter's DIY Thermocam is probably the most mature and well known.  He has done a great job and I pored over his code.  Damien Walsh's Leptonic is also really well done and works with the Lepton 3.5 as well.  Both Max and Damien were very gracious when I sent them various questions.

I decided to follow Max's lead and build a test platform using a Teensy 3.5 that I had (selected for the multiple SPI interfaces and copious RAM).  Unfortunately after soldering the Teensy to a Sparkfun breakout board I stressed the processor BGA package and made the board unreliable.  So I replaced it with a Teensy 3.2 hoping it would have enough resources to successfully interface to the Lepton.  It does, barely, and the next project log describes the test platform hardware.

teensy_schematic.pdf

Test platform schematic

Adobe Portable Document Format - 26.67 kB - 07/10/2018 at 19:11

Preview
Download

View all 18 components

  • tCam Web prototype

    Dan Julio02/25/2021 at 20:10 0 comments

    One long-standing goal I have had for my thermal cameras was to record events such as a wildlife in our driveway at nights (we think it's pretty busy out there) by detecting parts of the image in certain temperature ranges.  Sort of like a very souped up PIR detector.  The cameras would act like a security webcam.

    I've taken the first steps toward this vision with a prototype of tcam_web, a recording web server for tCam cameras.  It's written in xojo (which has some limitations described below) and can run on OS X, Windows and either x86 or ARM7 Linux hosts (my ultimate server will be a RPi 4).

    It can connect with one or more tCam cameras via a socket interface and one or more browsers via its web server interface.

    It can display an image from a camera or be configured to record images that meet certain detection criteria.  To do this it enables the cameras into streaming mode and analyzes the images it receives.  Right now the detection criteria is very simple.  It will record an image if at least certain percentage of pixels are within a specified temperature range.  I did try, unsuccessfully, to create a fast blob detector and hope to return to that at some point.  The detection criteria, along with some parameters limiting how many images can be stored based on a detection event, the streaming interval and camera configuration may be setup through the web interface. 

    The program also allows viewing images it has stored.  I intend for this application to be used in conjunction with the desktop tcam application.  The desktop application can be used to analyze example images to determine the selection criteria for tcam_web as well as view and analyze images it has recorded.

    The xojo development environment allowed me - someone who is not skilled in web techologies at all - to write this application fairly quickly and reusing code  originally created for the desktop application.  Awesome!  Unfortunately while xojo is very mature and effective for writing desktop applications, its new "Web 2.0" API is still slow and a little buggy.   The way it renders may be good for controls but is not so good for images.  I had hoped to be able to stream images from the camera to the web browser but without learning a bunch of web stuff and digging into their Web SDK that doesn't seem possible.  It takes a significant fraction of a second (and, apparently a couple of communication trips back and forth between the browser and application) to load a single image.  So right now you can just manually load one image at a time.  Also the code is currently single threaded and I suspect to handle more than a couple of cameras I will have to break that task into its own thread or process using their worker class.

    Eventually I want the application to be able to email or text (via twillio probably) an image when it has a detection event.

    Once the application has been debugged for a bit I will post binaries for the different platforms to github.

    I also plan to marry a tCam-Mini with my Solar MPPT charger for deployment in our lower driveway as soon as some snow melts.  I'll have the tCam-Mini report the charger's status as its own battery status.

  • tCam-Mini is a Group Gets campaign!

    Dan Julio02/18/2021 at 20:13 0 comments

    Interested in a tCam-Mini and don't want to wire one up yourself?  I've been working with the folks at Group Gets and tCam-Mini is a campaign!   If successful then they'll build a run of boards with their US-based CM for the backers.  If super successful then it might become an on-going product available in their store.

    Work continues on software support including a simple web server that can detect images with certain temperature ranges and record images and a python library to make interfacing with the tCam cameras easy.

  • Protecting the Lepton Outdoors

    Dan Julio02/14/2021 at 00:20 0 comments

    I have been researching lenses for thermal imaging cameras recently because one goal is to put a camera outdoors in a weatherproof enclosure streaming to software looking for heat signatures of wildlife crossing in front of the camera.  But because traditional glass and plastic materials are opaque to LWIR radiation the thermal imaging camera senses, other materials must be used.  Expensive materials!  Searching for lenses made with the materials Flir recommends in the data sheet  such as Germanium and Gallium Arsenide return results that cost far more than the Lepton itself.  Many of these lenses also can magnify or change the focal length, capabilities that may not be needed for simple applications where all one wants to do is protect the sensor.

    Further research took me to the EEVblog forums and some of the incredibly smart members there.  They were discussing this stuff way back in 2013 and 2014 if not sooner.  There I learned that materials like Polyolefin (shrink-wrap) film and even common kitchen materials such as cling wrap or plastic sandwich bags are very transparent to LWIR.

    A very quick experiment was somewhat promising.  My "super scientific" setup is shown below.

    The camera can still take a good image but computes a different radiometric temperature.  I suspect reflection from the camera's own internal heat from the plastic bag.  The more expensive lenses have anti-reflective coatings.

    A single layer of cling wrap resulted in a slightly smaller difference, less than 4°F.

    Obviously one wouldn't want to use cling wrap to protect against flying debris but with some kind of shroud or other structure could be quite useful for an outdoor thermal camera.

    I didn't test but read that Flir sells Polyolefin protectors.  I plan to obtain and test some Polyolefin sheets as well, preshrinking them over a frame that could be mounted in front of the sensor.

    Finally, an interesting link I found while researching.

    https://infraredtraininginstitute.com/thermograpahy-information/

  • Easier to build tCam-Mini

    Dan Julio02/07/2021 at 18:42 0 comments

    I found the TTGO V7 V1.4 ESP32 WROVER based development board long after I started tCam-Mini but it's a good way, along with a Lepton Breakout from Group Gets, to even more easily build a camera without having to make and load a PCB. The PSRAM is already part of the WROVER module simplifying the build considerably.  Just need to add a button, dual-color LED, pair of resistors and an electrolytic capacitor.

    Wiring is very straight forward.

    Two things to note about using the TTGO board. 

    1. It may have only 4 MB of Flash.  This is sufficient for the code but I use an 8 MB version on the PCB.  If you use the Espressif Programming tool to load precompiled firmware you will need to correctly specify the Flash size (see instructions in the "programming" directory on github).
    2. It may have an older revision ESP32 chip.  My TTGO board had a Rev 1 chip.  I specified a module for the PCB that has a Rev 3 chip and I built for that revision.  If the board won't boot (you can see an error message in the serial output) then you can try the version of the precompiled firmware I made for the Rev 1 chip also in the "firmware/precompiled" directory at the github repository.

  • tCam-Mini and Desktop App pushed to GitHub

    Dan Julio02/04/2021 at 22:02 0 comments

    Big news today.  I pushed the design and code for tCam mini to the github repository  (new section: ESP32).

    I've also been busy putting the finishing touches on the companion desktop application and binaries for it are also available.   Lots of bug fixes and three useful new features.

    1. Support for video files comprised of multiple radiometric images from the camera.  The tCam stream function can now stream at different rates enabling time lapse thermography.
    2. Radiometric markers to monitor temperature in more than one location in a scene.
    3. Graphing of the spotmeter and radiometric markers over time.

    The menu bar on the main window also got a slight layout change so it looks better on Linux (the app runs on OS X, Windows and Linux).

    Graphs can be made from real-time streaming data coming from a camera or they can be made from a video file.  In this case from the file "pi0_5sec_heading.tmjsn" that was created from a streaming session with an image every 5 seconds of a Pi 0 mounted on my Pi Platter heating up and then cooling down.

    Finally I added a U.FL connector to my prototype board and tested with an external antenna.  It provides better performance (higher streaming rates).  I can now get between 6-7 frame-per-second which may be near the limit of the ESP32 WiFi/network stack.

  • Software progress & Antenna questions

    Dan Julio01/17/2021 at 20:09 0 comments

    Some time over the holiday let me make quite a bit of progress with the tCam-mini firmware and desktop application software.  A little more testing and I'm ready to post the v1.0 tCam-mini firmware to github, along with the rev 1 hardware design.  Along with some cleanup the main change was to add the ability for the streaming enable command to include a mSec period between images and a maximum number of images.  This change was made to facilitate the upcoming graph function in the desktop application.

    The desktop application got a lot of work.  Aside from slight UI modifications and various bug fixes the main changes include

    • Support for radiometric data file vides and a new video file format (.tmjsn extension) that allows real-time playback of series of radiometric images.
    • Up to four radiometric markers allowing sampling the temperature in more places than just the spot meter alone.  These will also be used for the graphing function.
    • A more accurate frames-per-second readout for streaming and video playback.
    • Application and file icons
    • Filetype registration with the OS so double-clicking an image or video file opens the application and displays the file.
    • Drag and drop support.  Dragging an image or video file onto the application automatically opens it.

    I hope to be able to add a graphing function over the next couple of weeks that will complete the version 1.0 release of this application so that the temperature of multiple points can be plotted over time.

    I also did a lot of analysis of the firmware running on tCam-mini to try to understand why streaming performance varied so much.  I moved the Lepton readout task to be the only task running on the the APP (CPU 1) processor.  I also had to increase its priority but it is capable of keeping up with the Lepton's ~9 fps output rate with no problem.  The command processing and response tasks running alongside the WiFi stack on the PRO (CPU 0 processor do not require a lot of processing time.  The largest chunk of time is spent converting the binary radiometric and telemetry data to Base64 for inclusion the the json packet sent over the interface but that's only a few tens of milliseconds.

    The biggest variability comes from the system as it tries to send the data over the WiFi interface.  I see times from about 100-400 mSec per image (~55k).  This lead me to wonder about the antenna on the ESP32 module.  Sure enough things like position of the device in space and my finger touching the module made huge differences.

    The ESP32 hardware design guide has this to say about PCB layout.

    I had knowingly violated that spec with my layout to keep the board size down.  However chopping off the two little PCB wings designed to help mount the board had no effect - unlike my finger as can be seen in the almost double frame rate shown below.

    I am still at a loss but think that I will use an external antenna in the future connected via a U.FL connector on the ESP32 module.

    Up next is to try to finish version 1.0 of the desktop app, more testing and then upload everything to github in case anyone also wants to build a tCam-mini.  I will also have a couple of extra assembled PCBs that I'll either sell directly or on tindie.

  • Lepton flat-field measurements

    Dan Julio12/26/2020 at 18:56 0 comments

    The histogram output lets us look a little at the Lepton 3.5's internal operation.  I put the sensor just above a flat black metal surface (emissivity approaching 1.0) that has a reasonably constant temperature (~21.9°C at the time of measurement).

    The default operation is Radiometry enabled - TLinear output so that every pixel value from the Lepton is an absolute temperature with resolution of 0.01 K.  After letting the sensor run for about 30 minutes the output looks like this.

    There appears to be a temperature gradient across the surface of the sensor.  Perhaps the outer areas are prone to absorb radiant heat being generated by the Lepton and the ESP32 module on the other side of the board.  I won't claim that this setup is incredibly accurate but the temperature measured by the thermistor on the surface is different than the temperature reported by the Lepton by more than the +/- 5°C accuracy claimed in the spec sheet.  At some point I'd like to make a temperature controlled black-body radiator for some more precise accuracy measurements.  This might allow additional calibration factors to be applied on a per-sensor/camera basis for improved accuracy.

    The software's data processing algorithm scales the radiometric data to 8-bits to simplify display through the various look-up tables.  This accounts for some of the quantized look of this histogram data as the 120 possible radiometric values for a 1.2°C delta between minimum and maximum pixel temperatures is being displayed in a 256 point field.  However it appears the Lepton's internal algorithm may be slightly reducing the resolution as we only see 60 bin values.

    The placement of the ESP32 behind the Lepton maybe a source of error in this design if it causes a thermal gradient to appear across the camera.  I'm not sure this is the case however because over the course of many images I have not seen a constant gradient and usually the coldest part of the image is more-or-less centered.

    Interesting, also, to see what effect AGC has on the output data.  This image was taken only seconds after the one above.

  • tCam-Mini PCBs arrived!

    Dan Julio12/16/2020 at 18:10 0 comments

    Rev 1 of the PCBs arrived and seem to work great with one caveat.

    I built up two boards.  Both work, running the existing code, but one streams at a slower rate than the other.  With this design I moved to modules using the v3 ESP32 chip and even after recompiling for that version the slow-down on one module persisted.  I have noticed that one can get the FreeRTOS scheduler stuck in a pattern and I wonder if something like that is happening.  Or perhaps something with one module's wifi interface.  I'm going to add debugging code to see what part of the code is slowing down on one module.

    The board has the ability for the ESP32 to reset the Lepton and I'll add code for that and do some clean-up.  Then I should be able to post the code and hardware design on github.

    Probably also eventually rev the board to add a reset button (useful to restart while the serial monitor is attached) and tighten up the cutout for the LED to make assembly easier (or hopefully find a cheap side-looker RG SMT LED with lens to poke through a 3D printed enclosure (also have to design the enclosure...)

  • tCam-Mini

    Dan Julio11/30/2020 at 15:54 0 comments

    A socially distanced Thanksgiving holiday finally gave me some time to work on tCam.  Specifically I finished getting a new version that I’ve been working in spurts for a while called tCam-Mini debugged and running.  I also put to bed some bugs and memory leaks in the companion desktop application.

    My current idea is that there will be at least two models of tCam.  The full tCam with a local LCD display and file storage and tCam-Mini designed to be a small, simple streaming thermal imaging camera.  My work with the full tCam (now ported to the Fire Cam hardware for access to its 4-bit Micro-SD interface) shows that the ESP32 probably lacks sufficient horsepower to simultaneously display, record and stream over wifi the full 8.7 frames-per-second from the Lepton.  I plan to use two ESP32 chips for the full tCam and this will be the subject for a future project log.  However one ESP32 is sufficient for a small camera that forgoes the LCD and local file storage.

    All tCam cameras will work with the desktop application.  It will grow to include the ability to graph the temperature of selected points in the image over time (I intend to use this feature to monitor temperatures on circuit boards).  I’ve also started to contemplate another computer application.  This one will be a recording web server, hopefully with the ability to recognize some thermal signatures so the cameras can be left running and pictures or videos recorded when “interesting” things happen.  Access will be from any web browser.

    tCam-Mini acts as an access point by default but can be put on another network using the utility.  It can return single pictures or stream images.  It is based on the code from the full tCam code-base although some of the code was optimized for operation as a streaming camera.  The code base has also been ported to the IDF 4.0.2 release.  Although I am getting the full frame rate from the Lepton in one task, I can only stream 5-6 frames-per-second over wifi from another.  I hope further debugging will improve that number.

    The prototype is based around a Sparkfun Thing dev board.  Initially I hoped that I could get by without extra RAM but found I couldn’t fit the necessary buffering in the limited on-board space so I bodged a PSRAM on top of the module.

    Over the holiday I also laid out a circuit board for tCam-Mini and plan to mate it with a 3D printed enclosure.  It uses a WROVER ESP32 module and forgoes the Lepton breakout board to include the Lepton circuitry directly on board.  Aside from the Lepton, single unit price for the components and PCB is about $20.  I hope to build up a unit in December.

    Ultimately - after further optimization and cleanup - tCam-Mini will be open sourced.  I may try to sell them on Tindie but pricing will be tricky as I currently pay around $200 for each Lepton.  That makes the camera fairly expensive with even only a moderate mark-up.  Research on quantity pricing for the Lepton is in order…

  • tCam video

    Dan Julio06/24/2020 at 15:58 0 comments

    I entered tCam into a contest sponsored by Tech Briefs Media Group.  Here's a video made for the entry that shows the camera in operation.

View all 27 project logs

Enjoy this project?

Share

Discussions

leoshmu wrote 01/12/2021 at 03:23 point

Thank you for this excellent project. I am curious about pushing the limits of frame rate. Are you able to achieve a full 9Hz frame rate on the thermal video stream, or do you find that the frame rate is limited to less than 9Hz in practice. Could you imagine getting 2 leptons integrated together and offsetting the thermal stream so that they are off by 55ms, allowing for a combined 18hz signal? Perhaps something with a master clock circuit and then controlling each lepton separately based on the clock timing? Also I'm curious, why do you prefer the pure thermal board to the lepton v2 breakout board? 

  Are you sure? yes | no

Dan Julio wrote 02/04/2021 at 03:26 point

Hey Leo, thanks for the kind words.  I have been able to achieve the full 9Hz frame rate using the Beaglebone's PRU (in AGC mode) and with the ESP32 (both radiometric and AGC modes).  With the ESP32 I more-or-less dedicate one of the two CPUs to getting data out of the Lepton (and I use the VSYNC signal).  I will shortly be posting the code for tCam-Mini which is capable of reading the full frame rate (but not quite capable of sending that frame-rate over the WiFi).

Regarding your interesting idea of using two devices:  I'm not sure how to synchronize them.  I think there is a lot of internal processing that ends up affecting when the device is capable of sending a video packet - and things like a FFC also impact its ability to output a valid frame.  I suspect - but have no proof - that it is capable of a higher frame rate but held back in firmware (the sensor response time seems to be around 50 mSec).  I'm not up to it but I've wondered if someone could hack it.

I like the pure thermal board because it's smaller - I don't think there are any performance related differences.  Now, with tCam-Mini, I'm moving to directly mounting the sensor on my own boards.  

  Are you sure? yes | no

Ailsamly wrote 08/18/2020 at 10:30 point

Hi Dan,

I also plan to build a small thermal camera with a  suitable thermal sensor module, I want to connect it to a mobile platform and do building inspection (like water damage and air leakage and other thermal anomalies) .  The thermal sensor will be 5-10cm in front of the wall. I want to follow Max Ritter's project DIY-Thermocam. But I am not sure whether the resolution of Flir lepton can fulfull the requirements of mine. Do you have some ideas? Thank you!

Best,

Leyuan

  Are you sure? yes | no

Dan Julio wrote 12/26/2020 at 19:11 point

Hello Leyuan, I apologize for this tardy reply.  For some reason I never saw your post.  I think the Lepton should work from a distance of 5-10 cm.  However I can't answer your question about the resolution as I'm not sure what the characteristics of the things you are looking for are.  I can only suggest seeing if you could borrow a camera from someone and do some experimentation before committing to your own build.  Good luck though!  Sounds like a cool project.

  Are you sure? yes | no

Richard wrote 05/30/2020 at 16:15 point

Hi Dan,

According to Flir's datasheets, the Lepton uses 3.0V for IO functions, but the Teensy's on board Vreg runs at 3.3V. Did you do any level translation on the VOSPI and I2C interfaces or modify the Teensy to run at 3.0V, or just put 3.3V on the buses? I'm about to start a Lepton project of my own and want to be sure I don't kill my expensive sensor!

Thanks!

  Are you sure? yes | no

Dan Julio wrote 05/30/2020 at 16:49 point

Hey Richard,

You're right, they specify a 2.8-3.1 volt VDDIO.  But they also specify that the maximum on an IO pin is the lesser of VDDIO + 0.6 volts or 4.8 volts (table 18 - Absolute Maximum Ratings).  The Pure Thermal breakout boards have a 2.8 volt regulator feeding VDDIO so the maximum input voltage should be 3.4 volts.

To be honest I didn't pay attention to this spec when I first started.  I saw that the Pure Thermal boards were used with many different 3.3V SBCs so I jumped right in.  It wasn't until later I saw the specs you are referring to.

Although it doesn't seem like good engineering practice, I have run several Lepton 3.5 modules for long periods of time and this does not seem to change their functionality.  One could design their own breakout board for the Lepton and provide 3.1 volts to VDDIO which would lessen the over-voltage condition even more or provide resistor dividers on the SCLK and CSN signals (and pull SDA/SCL to the VDDIO rail).  As I work on what I hope is my final thermal imaging camera I am contemplating a full custom PCB (eliminating the breakout boards) and this is probably what I'll do, especially after your timely email :-)

  Are you sure? yes | no

Avi Cohen wrote 11/14/2019 at 11:45 point

Hi Dan, 

Is the code open to download?

  Are you sure? yes | no

Dan Julio wrote 11/14/2019 at 17:09 point

Hey Avi, 

yes, it's in the github repository.

https://github.com/danjulio/lepton

  Are you sure? yes | no

Sophi Kravitz wrote 07/10/2018 at 17:21 point

hi Dan! This is a cool project! This is happening pretty soon: https://hackaday.io/contest/159585-battery-powered-radiometric-lepton-contest

  Are you sure? yes | no

Dan Julio wrote 07/10/2018 at 18:09 point

Hi Sophi!  I saw that.  I'm not sure I'll have anything for the Lepton 2.5 but I will contribute a port of FLIR's LeptonSDKEmb32OEM library for the Teensy which might be applicable to the ESP32.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates