Close
0%
0%

Debugging on a Teensy, the open source way

There comes a time when debugging your code requires more than Serial.print();

Similar projects worth following
120 views
0 followers
While a Teensy is a seriously capable board for its form factor, there comes a point where a code running on it becomes too complex to use Serial.print for debugging. In those cases, a JTAG debug probe would be awesome.

This project investigates the use of an open source debug probe from blackmagic project on a Teensy 3.2, with the hope of extending the support to Teensy 4.

Intro

I am using Teensy 4 in one of my projects where I've seemingly hit a wall with debugging. Using printf statements messes up with the timings in the code, and I have no way of inspecting the state of the system at the time when things go wrong. There's no way of pausing the code, changing few variables and resuming the execution. I have tried this approach with some other boards that shipped with on-board debugger, and ended up wondering what would it take to get this to work on Teensy.

I have also tried a software solution to debugging, TeensyDebug, which essentially runs and GDB server directly on Teensy, together with your code. While this does a good job for most of the time, it seems to crash when my program dies, which is when I need it the most.

The only thing I believe could help is having a JTAG debugger to inspect the running code and be able to halt the MCU when a fault occurs.

Debugger?

This brings me to the second problem: I don't have a JTAG debugger. 

The ones I used in the past came as a part of eval kits (Texas Instruments Tiva series) and they will not work with other boards. So I need to get something else. A work colleague of mine recently showed me the blackmagic project, which is an open source debug probe, awesome! As an added bonus, they support tiva's debugger as a platform, so in theory I could reprogram on the eval kits I already have and have a general purpose debugger. Fantastic!

Getting the project to compile and run was a breeze. Now onto teensy.

Teensy 4?

Although I'd like this to work on Teensy 4, I would like to give it a try on Teensy 3 first. Reason for that is that I only have two Teensy 4 boards (one in use), but a lot more Teensy 3.2 boards I can spare, and the process sounds like it might be destructive - at least in the research phase.

Luckily, there's some good discussions around the internet on hacking in a debugger into teensy, that essentially outline the procedure or add extra bits of useful info:

Teensy 3 originally not supported by blackmagic

When starting this project, one of the hurdles was that MK20DX256, chip used in Teensy 3 was not originally supported by the blackmagic debug probe. However, I have since sent a pull request with support for it and it now works with the probe.

Wiring up Teensy 3.2

Teensy 3.2 features a bootloader chip (MKL02). That chip is what allows loading the firmware into Teensy chip using the Teensy loader PC program. It also ensure that no matter how bad you mess up your program, your board will always be programmable. Bootloader chip has access to the main chip (MK20DX256) via SWO interface. Since the board doesn't expose the pins in a convenient header, the only way is to tap into the connection between the bootloader and main chip.

Wiring it is a bit of a process and requires some thin wires and a steady hand. I ended up attaching all of the wires I could onto the via as they are easier to solder, and those that I couldn't directly onto the chip. Final result looks like this:

Pinout is as follows:

JTAG /SWOSignalMK20DX PinLocation Hint
TCK/SWCLKPTA022 (accessible from top of the 2 soldered vias)
TDIPTA123 (accessible from bottom of the 2 soldered vias)
TDO/tSWOPTA224 (8 from the top of the left side of the chip; accessible on chip only)
TMS/SWDIOPTA325 (9 from the top of the left side of the chip; accessible on chip only)
RSTRESET_b34 (exposed via 'R' test pad on the back side)

From what I understood online, both JTAG and SWO interface are usable on Teensy 3.2, so the choice of interface is mostly up to the probe hooked up to it.

In this configuration, bootloader...

Read more »

  • Debugging the debug session

    Vedran04/05/2024 at 16:28 0 comments

    I tried several other approaches with breakpoints

    • temporary breakpoint - fails when hit
    • software breakpoint added through GDB - not supported since the code running is in flash
    • adding a breakpoint, then stepping one instruction - code runs until a breakpoint is hit, then fails
    • adding a software breakpoint via __asm__("BKPT"); line fails as well

    It's likely something obvious I'm missing since I'm by no means knowledgeable in this area. So it's time to learn the basics of what's going on under the hood. I found a nice article to get me started on GDB, it's remote serial protocol (used to talk to a debug probe), and breakpoints in general: https://interrupt.memfault.com/blog/cortex-m-breakpoints


    With what I learned there, I could print out GDB commands and conclude that installing a breakpoint is not what causes a problem. Rather, failure happens consistently after a breakpoint is hit and GDB is trying to read general registers

    Following the printout:

    1. Code is paused (Ctrl-c)
    2. Temporary breakpoint is installed at in function loop() at line 23. ('tbreak ...')
    3. Request for resuming code execution ( '(gdb)c' )
    4. Request made to install a hardware breakpoint ($Z1) at address 0x47a
    5. Request made to continue execution ( '$c...' )
    6. MCU reports a breakpoint has been hit ('Packet received: T05')
    7. Request made to read registers( '$g...' )
    8. Request fails

    Playing with the debug firmware - A breakthrough!

    To further isolate the problem after a few days of hitting my head against the wall, I went on to poke at the blackmagic firmware. I could see above that the error always happens when requesting the registers after a breakpoint, during "$g" command. So I went in https://github.com/vedranMv/blackmagic/blob/kinetis_mk20/src/gdb_main.c#L117 file and tried to remove handling of that command from firmware in the way shown below:

    /* execute gdb remote command stored in 'pbuf'. returns immediately, no busy waiting. */
    int gdb_main_loop(target_controller_s *tc, char *pbuf, size_t pbuf_size, size_t size, bool in_syscall)
    {
        bool single_step = false;
    
        /* GDB protocol main loop */
        switch (pbuf[0]) {
        /* Implementation of these is mandatory! */
        case 'g': { /* 'g': Read general registers */
            ERROR_IF_NO_TARGET();
            const size_t reg_size = target_regs_size(cur_target);
            /*if (reg_size) {
                uint8_t *gp_regs = alloca(reg_size);
                target_regs_read(cur_target, gp_regs);
                gdb_putpacket(hexify(pbuf, gp_regs, reg_size), reg_size * 2U);
            } else {
                gdb_putpacketz("00");
            }*/
            gdb_putpacketz("00");
            break;
        }

    Flashing this on, the issue was no longer there. I could hit a breakpoint and resume from it without errors. The only problem now is that, the registers are no longer being updated.

    Conclusion

    After getting help from maintainers of blackmagic project, the issue turned out to be in GDB itself. My platformio project somehow ended up using a very old version of GDB that had a bug causing it to fail in special cases when reading registers from the MCU. Bug report in GDB's issue tracker was the final confirmation.

    In short, the bug was in the handling of a response GDB receives from a debug probe when querying target registers. If the registers look such that first three bytes look like an error code ('Exx....'), GDB would falsely interpret this as a probe encountering an error and would simply terminate the session. In some of the screenshots above, there was no "remote failure" as GDB states, there was just an unlucky situation that the first register returned in the reply had an unfortunate value that would be converted to "E09...".

    The fix was to simply upgrade GDB to a version where the bug is fixed, and since in platformio GDB is part of the compiler, that meant updating the compiler itself. To bump the compiler in platformIO, it's necessary to update the configuration with the following line:

    platform_packages = platformio/toolchain-gccarmnoneeabi@1.90301.200702
    

    Version can be any of the supported versions, as long as it's higher than 1.50401.190816...

    Read more »

  • GDB debugging

    Vedran04/03/2024 at 16:12 0 comments

    After establishing a connection, I could poke further at the GDB console. Using a simple blinky project for test

    #include <Arduino.h>
    
    // Set LED_BUILTIN if it is not defined by Arduino framework
    #ifndef LED_BUILTIN
        #define LED_BUILTIN 2
    #endif
    
    void setup()
    {
      // initialize LED digital pin as an output.
      pinMode(LED_BUILTIN, OUTPUT);
    }
    
    void loop()
    {
      // turn the LED on (HIGH is the voltage level)
      digitalWrite(LED_BUILTIN, HIGH);
      // wait for a second
      delay(1000);
      // turn the LED off by making the voltage LOW
      digitalWrite(LED_BUILTIN, LOW);
       // wait for a second
      delay(1000);
    }

    I got used to platformIO in VS code, so the example is made through Arduino framework. Before jumping into the platformIO's debugging integration, I thought to give it a test in CLI. Seems like there's no problems with loading the code...

    ..setting a break point, then running to it...

    However, continuing after a first breakpoint seems to fail. Before the failure, windows gives an audible sound that a USB device (Teensy, I assume) has been disconnected.

    Any attempts to resume the program, short of full restart ('run' command) fail.

    Attempt 2

    When pausing the program via ctrl-c during a run, and not using any breakpoints, sessions seems to work fine and there's no errors or device disconnects

    Few more tests isolated the error to the use of breakpoints, with the following observations:

    • Setting a breakpoint that will never be hit (e.g. in a setup() function after it has already run) will not produce an error. Unsetting that same breakpoint will
    • Setting a breakpoint in a place where it can be hit will cause an error seemingly before GDB reports that a breakpoint has occurred

  • Blackmagic and Teensy 3

    Vedran04/03/2024 at 15:57 0 comments

    Unfortunately, blackmagic currently doesn't support MK20 chip used by the Teensy.

    There's an old PR where someone tried to contribute to blackmagic by adding support for MK20, but sadly it never got in. That can be used as a base with the hope of pushing it upwards if it succeeds.

    From here on, I'm in a new area for me. Even though I've used several debuggers in development, my knowledge of what makes them tick and GDB console is pretty much none-existent. This is going to be fun!

  • Connecting a Teensy 3.2

    Vedran04/02/2024 at 19:15 0 comments

    Pin definitions for JTAG can be found in MK20 datasheet.

    Based on Teensy 3.2 schematics, the pins needed for JTAG are already connected to MKL0 bootloader chip. 

    This great article outlines a nice trick for using the JTAG lines, without having to physically cut the lines between teensy and a bootloader chip. Namely, there's a test point with a reset signal for for bootloader chip, preventing it from interfering with the signal lines.

    Testing it all

    Hooking it up to a blackmagic probe running my branch, I could confirm the approach works since the JTAG scan lists a device correctly. This simply starts GDB on a computer, connects to a blackmagic probe on a serial port and instructs the probe to scan JTAG for devices.

    And a test setup

View all 4 project logs

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates