Close

A side-trip to screen captures

A project log for bleXmang

BLE MITM Group. It's pronounced the same as "blanc mange", the dessert, but it's nothing like that.

wjcarpenterWJCarpenter 05/05/2022 at 16:270 Comments

Whoa! How did I get those crisp screen captures that were part of the last project log item? Do I have some kind of super-steady professional camera rig and awesome photo editing skills? No. In fact, the first draft of that project log did use mobile phone camera shots of the board. They looked pretty terrible.

I decided to see if I could get a literal pixel-perfect screen capture of my evolving UI screens. I deduced that the ST7789 controller has some kind of internal frame buffer holding the current screen image. Looking through the datasheet, it seemed like it would be possible to read out the color values of individual pixels, though there were some configuration settings that I would have to work my way through to figure out a lot of details. (For example, the graphics package is sending 16-bit RGB 565 color codes, but the controller has different ways of storing them in its frame buffer. What would I get back?)

It seemed at least feasible to be able to read back the entire image. The next question would be how to get that data off the device so that I could use it to construct some kind of standard graphical file. I pondered all kinds of exotic possibilities. For example, I could use WiFi to publish the data over MQTT to a willing subscriber. Or I could do something -- I'm not sure what -- over Bluetooth. All of those seem like a lot of bother just to get a few screen captures. A ready-made possibility would be to send it over the USB connection. That's already wired up in an ESP-IDF application to receive log output from the various ESP_LOGx macros. But the log output really wants to be textual, not binary, and there are various bits of ESP_LOGx boilerplate that would need to be trimmed away.

I started thinking about how to encode a rectangular array of pixel values into some unambiguous text format. For example, I could send each pixel as an (x,y) point and a hexadecimal color value. I could then write some scripting that would take that as input and create a PNG file. (Note to self: research PNG file format.) If only there were a graphics file format that was already text based, that would simplify my life.

Hey, wait a minute! There is such a thing. It's Scalable Vector Graphics (SVG). In normal circumstances, SVG is a poor choice for pixel-based graphics. The whole reason for SVG is to avoid that per-pixel lock-in. But it is actually possible to do pixel things, with the standard approach being to represent them as 1x1 rectangles. Given a clunky SVG representation of a screen, there are tons of tools for converting it into a PNG, JPEG, BMP, or some other graphics formats.

So, now all I have to do is figure out how to read the pixels out of the ST7789 frame buffer (all the info I should need is in the device datasheet), encode each one as an SVG rectangle, and spit it out via ESP_LOGx statements. The scripting on the computer side to carve out the SVG should be straightforward.

Is there a simpler way? Once I started thinking about SVG, I started thinking how it was composed of logical entities: circles, lines, rectangles, etc. Come to think of it, so are the operations in the esp-idf-st7789 library that I'm using. That library is a well-organized single C source file that already has a dependency on being inside the ESP-IDF ecosystem. I decided to modify that library to optionally emit the equivalent SVG graphics via ESP_LOGx statements. (Most of my UI is text, and the graphics library renders text by placing individual pixels. SVG also has text primitives, but using them would break the goal of pixel-perfect representation.)

I sent my changes to the graphics library to the upstream author via a pull request.

The process for using this technique is straightforward. Modify the application code to make calls to start and end emitting SVG. Run that code on the device while capturing the ESP_LOGx output. Use ordinary text tools to carve the SVG part out of the overall captured logging. Go back and disable the SVG logging in the application code (it adds quite a performance burden if there are a lot of things like text painted a pixel at a time). Here is a note I left in the bleXmang code to remind myself how to do this (on Linux). It also depends on some additional logging I do between screens (you can see those details in the code).

/*
 To get all of the SVGs in a single HTML page, you can capture the log output like this on Linux
 (turn off ANSI colors in the ESP-IDF project config settings (Component confg > Log output)):

    idf.py monitor | grep SVG | cut -d ' ' -f 4- | tee /tmp/svg.html

 It won't have the end  tag, but browsers don't care about that.

 To get the individual SVGs as separate chunks, be in an empty directory and do this:

    cat /tmp/svg.html | grep -v 'div>' | grep -v '' | csplit -f svg -b '%02d.svg' --suppress-matched -z - '/cut-cut-cut-cut/' '{*}'

 That will give you a series of files with names like "svg00.svg", "svg01.svg", etc.

 The graphics library is operating in portrait mode. To get landscape mode, rotate 90 CCW.
 With a recent version if inkscape, you can do it like this:

   inkscape --with-gui --verb="EditSelectAll;SelectionGroup;ObjectRotate90CCW;FitCanvasToSelection;FileSave;FileQuit" svg*.svg

 and then convert to PNG like this:

   inkscape --export-type="png" svg*.svg
*/

Discussions