Close
0%
0%

Lepton 3.5 Thermal Imaging Camera

Documenting my experiments with the FLIR Lepton 3.5 thermal imaging camera.

Similar projects worth following
I bought a FLIR Lepton 3.5 mounted on the Pure Engineering breakout after using a friend's thermal imaging camera to analyze heat generation on a PCB I designed. The Lepton 3.5 can output absolute temperature data (Radiometric) at 160x120 pixel resolution. I decided a thermal imaging camera would be a good tool to have and decided to build my own. I have started with a Teensy 3.2 based test platform but eventually want to turn a Beaglebone Black with 7" LCD cape into a full-featured, networked, camera using the PRUs to handle the real-time video SPI feed from the Lepton. I hope the documentation in this project is helpful to others who might also want to play with these amazing devices.

The long-term goal is to create a capable thermal imaging camera using the Beaglebone Black and a 7" LCD cape as the platform matching some of the features of high end commercial products.  However it soon became obvious that I'd need a simpler platform to learn how to use the Lepton module when I started reading the documentation and playing with the various demo codebases.  It is a very capable device with a moderately complex interface, both firmware and hardware.  Although the device has good default settings I found enabling some features wasn't well documented and the video SPI interface (VoSPI) challenging to implement due to its real-time constraints.


There are a lot of other great projects online to help get going with the FLIR sensors.  Pure Engineering is to be commended for making these devices available to makers and provides a wealth of code examples, many designed to work with the previous Lepton models.  Max Ritter's DIY Thermocam is probably the most mature and well known.  He has done a great job and I pored over his code.  Damien Walsh's Leptonic is also really well done and works with the Lepton 3.5 as well.  Both Max and Damien were very gracious when I sent them various questions.

I decided to follow Max's lead and build a test platform using a Teensy 3.5 that I had (selected for the multiple SPI interfaces and copious RAM).  Unfortunately after soldering the Teensy to a Sparkfun breakout board I stressed the processor BGA package and made the board unreliable.  So I replaced it with a Teensy 3.2 hoping it would have enough resources to successfully interface to the Lepton.  It does, barely, and the next project log describes the test platform hardware.

teensy_schematic.pdf

Test platform schematic

Adobe Portable Document Format - 26.67 kB - 07/10/2018 at 19:11

Preview
Download

View all 18 components

  • [Finally] starting to look at accuracy!

    Dan Julio03/29/2021 at 18:56 0 comments

    One of my many goals with this project is to characterize the Lepton's performance and see if I can improve the stated +/- 5 - 10°C accuracy.  I have finally started to make some progress in that direction.

    The Black Body radiator is a gold standard for measuring an imager's performance and providing a known temperature for calibration.   Unfortunately, commercial units are often very expensive limiting their availability to hobbyists.  Fortunately older devices can be had - and repurposed - from places like ebay.

    A collection of very knowledgeable thermal imaging enthusiast exists on Dave Jones' wonderful eevblog website and a couple of them put me onto the idea of repurposing an old calibrator designed for in-ear temperature monitors.

    The IVAC 9000 probably sold, originally, for thousands of dollars but go for a fraction of the price in the used market.  It contains two radiators precisely temperature controlled using Thermoelectric Controllers (peltier) at 26°C and 38°C.  The user plugs a cable into the battery compartment of the thermometer and sticks its tip into a plastic receptacle in front of one of the radiators and the device will read the thermometer and upload appropriate calibration data.  The two temperatures are picked for calibrating a device designed to measure human temperature but aren't really far enough apart for a thermal imaging camera visualizing a much larger range.

    The radiators are made up of two blocks of aluminum with the TEC sandwiched in between.  The front block is painted with a very high emissivity (0.97-0.99) paint and contains a thermistor.  The IVAC 9000 controller uses an H-bridge to control heating and cooling of the TEC to maintain a very precise temperature. 

    Originally I was going to try to replace the controller with my own but another smart hacker experimented with simply adding a series resistance to one of the thermistors to trick the existing controller into maintaining a different set point.  Using a multi-turn pot allows dialing in a higher temperature on the "high" side of the unit.  Unfortunately adding a parallel resistor to the "low" side thermistor didn't successfully result in a lower temperature so there's probably some limit in the firmware or the physics of cooling the low side.  I decided to take this tact as it was MUCH easier and to also add a precision temperature monitor that I could access remotely via Bluetooth for use in a calibration process if possible.

    I stripped off the front panel to get a view of the entire radiator (the two circular regions in the photo above) and put the old control panel PCB inside the unit.  The temperature monitor goes in front.  My unit can sustain 26 and 64°C as the two temperatures which, at least, is a start.

    Measuring some Leptons

    With the unit up and running it was time to see how some Leptons performed.  I took some quick readings of two tCam-Minis, the tCam prototype and a Flirone Pro connected to my phone.

    All cameras were placed about 10cm from the radiator with the spotmeter in the center of the image aimed at about the center of the radiator.  Emissivity for the tCam cameras was set to 0.97 and the Flirone Pro set to "Matte" (they don't give the user precise control over the actual value).   The cameras were running for at least 30 minutes before taking a measurement.

    CameraLow-side (26°C)High-side (64°C)
    tCam-Mini-CB4927.467.8
    tCam-Mini-B13225.464.1
    tCam-62F125.163.3
    Flirone Pro26.767.2

    Exciting to finally get data and to see it make some sense.  All the cameras were well within the specified range but it will be interesting to see if a calibration can be figured out.  To do that will may take the ability to more easily measure at a larger range of temperatures and to that end I'm also working on another radiator using an off-the-shelf TEC controller.

    First distance...

    Read more »

  • Laser-cut front/back plates for tCam-Mini

    Dan Julio03/29/2021 at 02:40 1 comment

    A super simple set of SVG and DXF drawings have been uploaded to my github that can be used on a laser cutter to cut a pair of front and back plates out of a piece of plastic.  They can be mounted with ~8mm stand-offs for the front plate and ~5mm/ stand-offs for the rear plate.

  • Rev 2 PCB testing

    Dan Julio03/11/2021 at 19:19 0 comments

    I received and tested the Rev 2 PCB design.  The very minor revision was an attempt to improve Wifi performance and make it easier to mount the LED.  As shown below Wifi performance is about the same.

    I tested Wifi performance with three setups.  One with the tCam-Mini acting as an Access Point and my laptop connected to it in the same room.  A second with tCam-Mini configured as a client connected to a OpenWrt router in the same room and a third with the camera configured as a client connected to a Netgear router in a different room.  I streamed data for three minutes to compute an average fps and I ran each test twice.  The average values are shown below, including a board with an external antenna.  There is some variability, even with three minute runs, due to Lepton FFCs that occurred while measuring.

    SetupRev 1 - internal antennaRev 2 - internal antennaRev 1 - external antenna
    Access Point4.36 fps4.48 fps5.94 fps
    Client -  OpenWrt router4.68 fps4.19 fps6.34 fps
    Client - Netgear router4.86 fps4.89 fps6.05 fps

    Clearly an external antenna improves performance.  Interestingly the Netgear router in another room performed better than the Openwrt (fairly old mips-based chipset) in the same room.

    Firmware Revisions

    Rev 1.2 firmware fixes two issues that have come up during testing and board bring-up.  The first was a bug where the board would not connect to an AP if its first attempt failed (it would reconnect later with no problems).  The second was a hardware issue where I found the Lepton was not being left in a good condition if its external crystal oscillator took too long to startup.  I found this testing alternate crystals since currently the crystal I specified is unobtanium. 

    I have an idea about how to improve the camera's performance in firmware.  Right now a single task handles converting the Lepton data into a json string with Base64 packed data and pushing into the ESP32's network stack for transmission.  If pushing the data causes the task to block then I may be able to increase performance by splitting the process into two tasks so the next frame can be converted to a json string while the previous frame is loaded for transmission.

  • tCam Web prototype

    Dan Julio02/25/2021 at 20:10 0 comments

    One long-standing goal I have had for my thermal cameras was to record events such as a wildlife in our driveway at nights (we think it's pretty busy out there) by detecting parts of the image in certain temperature ranges.  Sort of like a very souped up PIR detector.  The cameras would act like a security webcam.

    I've taken the first steps toward this vision with a prototype of tcam_web, a recording web server for tCam cameras.  It's written in xojo (which has some limitations described below) and can run on OS X, Windows and either x86 or ARM7 Linux hosts (my ultimate server will be a RPi 4).

    It can connect with one or more tCam cameras via a socket interface and one or more browsers via its web server interface.

    It can display an image from a camera or be configured to record images that meet certain detection criteria.  To do this it enables the cameras into streaming mode and analyzes the images it receives.  Right now the detection criteria is very simple.  It will record an image if at least certain percentage of pixels are within a specified temperature range.  I did try, unsuccessfully, to create a fast blob detector and hope to return to that at some point.  The detection criteria, along with some parameters limiting how many images can be stored based on a detection event, the streaming interval and camera configuration may be setup through the web interface. 

    The program also allows viewing images it has stored.  I intend for this application to be used in conjunction with the desktop tcam application.  The desktop application can be used to analyze example images to determine the selection criteria for tcam_web as well as view and analyze images it has recorded.

    The xojo development environment allowed me - someone who is not skilled in web techologies at all - to write this application fairly quickly and reusing code  originally created for the desktop application.  Awesome!  Unfortunately while xojo is very mature and effective for writing desktop applications, its new "Web 2.0" API is still slow and a little buggy.   The way it renders may be good for controls but is not so good for images.  I had hoped to be able to stream images from the camera to the web browser but without learning a bunch of web stuff and digging into their Web SDK that doesn't seem possible.  It takes a significant fraction of a second (and, apparently a couple of communication trips back and forth between the browser and application) to load a single image.  So right now you can just manually load one image at a time.  Also the code is currently single threaded and I suspect to handle more than a couple of cameras I will have to break that task into its own thread or process using their worker class.

    Eventually I want the application to be able to email or text (via twillio probably) an image when it has a detection event.

    Once the application has been debugged for a bit I will post binaries for the different platforms to github.

    I also plan to marry a tCam-Mini with my Solar MPPT charger for deployment in our lower driveway as soon as some snow melts.  I'll have the tCam-Mini report the charger's status as its own battery status.

  • tCam-Mini is a Group Gets campaign!

    Dan Julio02/18/2021 at 20:13 0 comments

    Interested in a tCam-Mini and don't want to wire one up yourself?  I've been working with the folks at Group Gets and tCam-Mini is a campaign!   If successful then they'll build a run of boards with their US-based CM for the backers.  If super successful then it might become an on-going product available in their store.

    Work continues on software support including a simple web server that can detect images with certain temperature ranges and record images and a python library to make interfacing with the tCam cameras easy.

  • Protecting the Lepton Outdoors

    Dan Julio02/14/2021 at 00:20 0 comments

    I have been researching lenses for thermal imaging cameras recently because one goal is to put a camera outdoors in a weatherproof enclosure streaming to software looking for heat signatures of wildlife crossing in front of the camera.  But because traditional glass and plastic materials are opaque to LWIR radiation the thermal imaging camera senses, other materials must be used.  Expensive materials!  Searching for lenses made with the materials Flir recommends in the data sheet  such as Germanium and Gallium Arsenide return results that cost far more than the Lepton itself.  Many of these lenses also can magnify or change the focal length, capabilities that may not be needed for simple applications where all one wants to do is protect the sensor.

    Further research took me to the EEVblog forums and some of the incredibly smart members there.  They were discussing this stuff way back in 2013 and 2014 if not sooner.  There I learned that materials like Polyolefin (shrink-wrap) film and even common kitchen materials such as cling wrap or plastic sandwich bags are very transparent to LWIR.

    A very quick experiment was somewhat promising.  My "super scientific" setup is shown below.

    The camera can still take a good image but computes a different radiometric temperature.  I suspect reflection from the camera's own internal heat from the plastic bag.  The more expensive lenses have anti-reflective coatings.

    A single layer of cling wrap resulted in a slightly smaller difference, less than 4°F.

    Obviously one wouldn't want to use cling wrap to protect against flying debris but with some kind of shroud or other structure could be quite useful for an outdoor thermal camera.

    I didn't test but read that Flir sells Polyolefin protectors.  I plan to obtain and test some Polyolefin sheets as well, preshrinking them over a frame that could be mounted in front of the sensor.

    Finally, an interesting link I found while researching.

    https://infraredtraininginstitute.com/thermograpahy-information/

  • Easier to build tCam-Mini

    Dan Julio02/07/2021 at 18:42 0 comments

    I found the TTGO V7 V1.4 ESP32 WROVER based development board long after I started tCam-Mini but it's a good way, along with a Lepton Breakout from Group Gets, to even more easily build a camera without having to make and load a PCB. The PSRAM is already part of the WROVER module simplifying the build considerably.  Just need to add a button, dual-color LED, pair of resistors and an electrolytic capacitor.

    Wiring is very straight forward.

    Two things to note about using the TTGO board. 

    1. It may have only 4 MB of Flash.  This is sufficient for the code but I use an 8 MB version on the PCB.  If you use the Espressif Programming tool to load precompiled firmware you will need to correctly specify the Flash size (see instructions in the "programming" directory on github).
    2. It may have an older revision ESP32 chip.  My TTGO board had a Rev 1 chip.  I specified a module for the PCB that has a Rev 3 chip and I built for that revision.  If the board won't boot (you can see an error message in the serial output) then you can try the version of the precompiled firmware I made for the Rev 1 chip also in the "firmware/precompiled" directory at the github repository.

  • tCam-Mini and Desktop App pushed to GitHub

    Dan Julio02/04/2021 at 22:02 0 comments

    Big news today.  I pushed the design and code for tCam mini to the github repository  (new section: ESP32).

    I've also been busy putting the finishing touches on the companion desktop application and binaries for it are also available.   Lots of bug fixes and three useful new features.

    1. Support for video files comprised of multiple radiometric images from the camera.  The tCam stream function can now stream at different rates enabling time lapse thermography.
    2. Radiometric markers to monitor temperature in more than one location in a scene.
    3. Graphing of the spotmeter and radiometric markers over time.

    The menu bar on the main window also got a slight layout change so it looks better on Linux (the app runs on OS X, Windows and Linux).

    Graphs can be made from real-time streaming data coming from a camera or they can be made from a video file.  In this case from the file "pi0_5sec_heading.tmjsn" that was created from a streaming session with an image every 5 seconds of a Pi 0 mounted on my Pi Platter heating up and then cooling down.

    Finally I added a U.FL connector to my prototype board and tested with an external antenna.  It provides better performance (higher streaming rates).  I can now get between 6-7 frame-per-second which may be near the limit of the ESP32 WiFi/network stack.

  • Software progress & Antenna questions

    Dan Julio01/17/2021 at 20:09 0 comments

    Some time over the holiday let me make quite a bit of progress with the tCam-mini firmware and desktop application software.  A little more testing and I'm ready to post the v1.0 tCam-mini firmware to github, along with the rev 1 hardware design.  Along with some cleanup the main change was to add the ability for the streaming enable command to include a mSec period between images and a maximum number of images.  This change was made to facilitate the upcoming graph function in the desktop application.

    The desktop application got a lot of work.  Aside from slight UI modifications and various bug fixes the main changes include

    • Support for radiometric data file vides and a new video file format (.tmjsn extension) that allows real-time playback of series of radiometric images.
    • Up to four radiometric markers allowing sampling the temperature in more places than just the spot meter alone.  These will also be used for the graphing function.
    • A more accurate frames-per-second readout for streaming and video playback.
    • Application and file icons
    • Filetype registration with the OS so double-clicking an image or video file opens the application and displays the file.
    • Drag and drop support.  Dragging an image or video file onto the application automatically opens it.

    I hope to be able to add a graphing function over the next couple of weeks that will complete the version 1.0 release of this application so that the temperature of multiple points can be plotted over time.

    I also did a lot of analysis of the firmware running on tCam-mini to try to understand why streaming performance varied so much.  I moved the Lepton readout task to be the only task running on the the APP (CPU 1) processor.  I also had to increase its priority but it is capable of keeping up with the Lepton's ~9 fps output rate with no problem.  The command processing and response tasks running alongside the WiFi stack on the PRO (CPU 0 processor do not require a lot of processing time.  The largest chunk of time is spent converting the binary radiometric and telemetry data to Base64 for inclusion the the json packet sent over the interface but that's only a few tens of milliseconds.

    The biggest variability comes from the system as it tries to send the data over the WiFi interface.  I see times from about 100-400 mSec per image (~55k).  This lead me to wonder about the antenna on the ESP32 module.  Sure enough things like position of the device in space and my finger touching the module made huge differences.

    The ESP32 hardware design guide has this to say about PCB layout.

    I had knowingly violated that spec with my layout to keep the board size down.  However chopping off the two little PCB wings designed to help mount the board had no effect - unlike my finger as can be seen in the almost double frame rate shown below.

    I am still at a loss but think that I will use an external antenna in the future connected via a U.FL connector on the ESP32 module.

    Up next is to try to finish version 1.0 of the desktop app, more testing and then upload everything to github in case anyone also wants to build a tCam-mini.  I will also have a couple of extra assembled PCBs that I'll either sell directly or on tindie.

  • Lepton flat-field measurements

    Dan Julio12/26/2020 at 18:56 0 comments

    The histogram output lets us look a little at the Lepton 3.5's internal operation.  I put the sensor just above a flat black metal surface (emissivity approaching 1.0) that has a reasonably constant temperature (~21.9°C at the time of measurement).

    The default operation is Radiometry enabled - TLinear output so that every pixel value from the Lepton is an absolute temperature with resolution of 0.01 K.  After letting the sensor run for about 30 minutes the output looks like this.

    There appears to be a temperature gradient across the surface of the sensor.  Perhaps the outer areas are prone to absorb radiant heat being generated by the Lepton and the ESP32 module on the other side of the board.  I won't claim that this setup is incredibly accurate but the temperature measured by the thermistor on the surface is different than the temperature reported by the Lepton by more than the +/- 5°C accuracy claimed in the spec sheet.  At some point I'd like to make a temperature controlled black-body radiator for some more precise accuracy measurements.  This might allow additional calibration factors to be applied on a per-sensor/camera basis for improved accuracy.

    The software's data processing algorithm scales the radiometric data to 8-bits to simplify display through the various look-up tables.  This accounts for some of the quantized look of this histogram data as the 120 possible radiometric values for a 1.2°C delta between minimum and maximum pixel temperatures is being displayed in a 256 point field.  However it appears the Lepton's internal algorithm may be slightly reducing the resolution as we only see 60 bin values.

    The placement of the ESP32 behind the Lepton maybe a source of error in this design if it causes a thermal gradient to appear across the camera.  I'm not sure this is the case however because over the course of many images I have not seen a constant gradient and usually the coldest part of the image is more-or-less centered.

    Interesting, also, to see what effect AGC has on the output data.  This image was taken only seconds after the one above.

View all 30 project logs

Enjoy this project?

Share

Discussions

leoshmu wrote 01/12/2021 at 03:23 point

Thank you for this excellent project. I am curious about pushing the limits of frame rate. Are you able to achieve a full 9Hz frame rate on the thermal video stream, or do you find that the frame rate is limited to less than 9Hz in practice. Could you imagine getting 2 leptons integrated together and offsetting the thermal stream so that they are off by 55ms, allowing for a combined 18hz signal? Perhaps something with a master clock circuit and then controlling each lepton separately based on the clock timing? Also I'm curious, why do you prefer the pure thermal board to the lepton v2 breakout board? 

  Are you sure? yes | no

Dan Julio wrote 02/04/2021 at 03:26 point

Hey Leo, thanks for the kind words.  I have been able to achieve the full 9Hz frame rate using the Beaglebone's PRU (in AGC mode) and with the ESP32 (both radiometric and AGC modes).  With the ESP32 I more-or-less dedicate one of the two CPUs to getting data out of the Lepton (and I use the VSYNC signal).  I will shortly be posting the code for tCam-Mini which is capable of reading the full frame rate (but not quite capable of sending that frame-rate over the WiFi).

Regarding your interesting idea of using two devices:  I'm not sure how to synchronize them.  I think there is a lot of internal processing that ends up affecting when the device is capable of sending a video packet - and things like a FFC also impact its ability to output a valid frame.  I suspect - but have no proof - that it is capable of a higher frame rate but held back in firmware (the sensor response time seems to be around 50 mSec).  I'm not up to it but I've wondered if someone could hack it.

I like the pure thermal board because it's smaller - I don't think there are any performance related differences.  Now, with tCam-Mini, I'm moving to directly mounting the sensor on my own boards.  

  Are you sure? yes | no

Ailsamly wrote 08/18/2020 at 10:30 point

Hi Dan,

I also plan to build a small thermal camera with a  suitable thermal sensor module, I want to connect it to a mobile platform and do building inspection (like water damage and air leakage and other thermal anomalies) .  The thermal sensor will be 5-10cm in front of the wall. I want to follow Max Ritter's project DIY-Thermocam. But I am not sure whether the resolution of Flir lepton can fulfull the requirements of mine. Do you have some ideas? Thank you!

Best,

Leyuan

  Are you sure? yes | no

Dan Julio wrote 12/26/2020 at 19:11 point

Hello Leyuan, I apologize for this tardy reply.  For some reason I never saw your post.  I think the Lepton should work from a distance of 5-10 cm.  However I can't answer your question about the resolution as I'm not sure what the characteristics of the things you are looking for are.  I can only suggest seeing if you could borrow a camera from someone and do some experimentation before committing to your own build.  Good luck though!  Sounds like a cool project.

  Are you sure? yes | no

Richard wrote 05/30/2020 at 16:15 point

Hi Dan,

According to Flir's datasheets, the Lepton uses 3.0V for IO functions, but the Teensy's on board Vreg runs at 3.3V. Did you do any level translation on the VOSPI and I2C interfaces or modify the Teensy to run at 3.0V, or just put 3.3V on the buses? I'm about to start a Lepton project of my own and want to be sure I don't kill my expensive sensor!

Thanks!

  Are you sure? yes | no

Dan Julio wrote 05/30/2020 at 16:49 point

Hey Richard,

You're right, they specify a 2.8-3.1 volt VDDIO.  But they also specify that the maximum on an IO pin is the lesser of VDDIO + 0.6 volts or 4.8 volts (table 18 - Absolute Maximum Ratings).  The Pure Thermal breakout boards have a 2.8 volt regulator feeding VDDIO so the maximum input voltage should be 3.4 volts.

To be honest I didn't pay attention to this spec when I first started.  I saw that the Pure Thermal boards were used with many different 3.3V SBCs so I jumped right in.  It wasn't until later I saw the specs you are referring to.

Although it doesn't seem like good engineering practice, I have run several Lepton 3.5 modules for long periods of time and this does not seem to change their functionality.  One could design their own breakout board for the Lepton and provide 3.1 volts to VDDIO which would lessen the over-voltage condition even more or provide resistor dividers on the SCLK and CSN signals (and pull SDA/SCL to the VDDIO rail).  As I work on what I hope is my final thermal imaging camera I am contemplating a full custom PCB (eliminating the breakout boards) and this is probably what I'll do, especially after your timely email :-)

  Are you sure? yes | no

Avi Cohen wrote 11/14/2019 at 11:45 point

Hi Dan, 

Is the code open to download?

  Are you sure? yes | no

Dan Julio wrote 11/14/2019 at 17:09 point

Hey Avi, 

yes, it's in the github repository.

https://github.com/danjulio/lepton

  Are you sure? yes | no

Sophi Kravitz wrote 07/10/2018 at 17:21 point

hi Dan! This is a cool project! This is happening pretty soon: https://hackaday.io/contest/159585-battery-powered-radiometric-lepton-contest

  Are you sure? yes | no

Dan Julio wrote 07/10/2018 at 18:09 point

Hi Sophi!  I saw that.  I'm not sure I'll have anything for the Lepton 2.5 but I will contribute a port of FLIR's LeptonSDKEmb32OEM library for the Teensy which might be applicable to the ESP32.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates