Since the last update, I've continued making bits of sporadic process.

I've:

• Used the FIT SDK code generator to get some C code that can write .fit files
• Written a network service to handle WiFi connectivity
• Written a logging service to write GPS data to a .fit file
• Put it all together and generated some .fit GPS tracks.

So I'll break those down a bit more here.

FIT SDK

The FIT (Flexible & Interoperable data Transfer, apparently) file format is a very flexible, openly documented data format used by a bunch of different GPS/activity tracker products, with the main driving force seeming to be Garmin (perhaps via their acquisition of Dynastream?).

There's an SDK available for download, which includes the various specifications and a tool for generating C/C++/C# and Java code to encode and decode fit files: https://www.thisisant.com/developer/resources/downloads/

As far as flexible data interchange formats go, FIT is fairly sane and well thought out.

The whole file format is based on "messages", where a message is just a set of fields. Each field has a type, which can either be a base type like "uint8" or "string", or a more specific enumerated type such as "WorkoutStep" which uses one of the base-types as its underlying representation. All of these messages and types are defined in a big spreadsheet in the SDK.

A FIT file consists of a header stating what kind of file it is ("Settings", "Device", "Activity" etc), and a series of messages with the actual content.

The file protocol allows for only encoding the fields which you actually care about, so even though the "Lap" message which stores data about a lap has 114 fields defined - from start time and position to "avg_saturated_hemoglobin_percent" - you only actually have to encode the fields you care about, and all decoders will gracefully handle the difference.

While I'm usually pretty cynical about proprietary file formats, I really do think FIT's a pretty elegant and effective design, and it's worth having a read through the documentation in the SDK.

Code Generator

FIT has the concept of a "profile" which is the subset of messages and fields that a particular device supports, and the code generator can generate suitable code for that profile. So, in order to generate just the code and definitions that I need for my particular project, I had to write (modify) a "profile.csv" file, which is the input to the code generator. In the .csv file you state which messages and fields you want, and then the Windows-only code generator will spit out C/C++/C#/Java code for it for you.

I wrote a very thin wrapper over the top of the generated code, which gives me just enough to create the FIT files I need. You can see the result here: https://github.com/usedbytes/tbeam-fit

Network Service

The next piece is the network service, which interfaces with the ESP32 WiFi and HTTP request libraries.

It's got two main responsibilities:

• Monitor the various network events from the ESP32 network stack, and report the relevant ones to other services which register for notifications, such as "network connected"
• Provide an interface to perform HTTP GET/POST requests

I'm not entirely happy with the request interface, but it does provide a pretty flexible way to perform network transactions, which should work fine for this project and is probably applicable to other projects, too.

The main interface is this structure:

struct network_txn {
    struct service *sender;

    esp_http_client_config_t cfg;

    // Content-Type header
    const char *content_type;

    // Callbacks for performing send and receive
    network_txn_send_cb send_cb;
    network_txn_receive_cb receive_cb;

    // Filled in by transaction handler
    esp_err_t err;
    int result;
};

 The caller fills in the 'cfg' structure with things like the URL, and provides two optional callbacks: 'send_cb' and 'receive_cb', then it sends a pointer to this request structure to the network service.

The network service will open a HTTP connection, and then call 'send_cb', which should send whatever data it wants to the server, which it can do in chunks. By doing this via a callback, I can have lots of flexibility for where the data comes from. For instance there might be a request were we just send a static string, in which case 'send_cb' would something like this:

esp_err_t string_send_cb(struct network_txn *base, network_send_chunk_fn send_chunk, esp_http_client_handle_t client)
{
    esp_err_t err = send_chunk(client, "Send my string", strlen("Send my string"));
    if (err != ESP_OK) {
        return err;
    }

    // We send an empty chunk to say that we're done
    err = send_chunk(client, NULL, 0);
    if (err != ESP_OK) {
        return err;
    }

    return ESP_OK;
}

Or, we might want to send a file, in which case the callback could read sections of the file into a local buffer, and then send it over the network in chunks without ever having the load the whole thing into memory.

esp_err_t network_file_post_send(struct network_txn *base, network_send_chunk_fn send_chunk,
                   esp_http_client_handle_t client)
{
    struct network_file_post_txn *txn = (struct network_file_post_txn *)base;
    char buf[512];
    esp_err_t err;
    int ret;

    while ((ret = fread(buf, 1, sizeof(buf), txn->fp)) > 0) {
        esp_err_t err = send_chunk(client, buf, ret);
        if (err != ESP_OK) {
            break;
        }
    }
    if (!feof(txn->fp) || ferror(txn->fp)) {
        err = ESP_FAIL;
        goto done;
    }

    err = send_chunk(client, NULL, 0);
    if (err != ESP_OK) {
        goto done;
    }

done:
    fclose(txn->fp);
    return err;
}

This would also be important if I implement heatshrink compression, as I would decompress on the fly before sending to the network.

Logging Service

The last new service is the logging service, which is currently pretty hacked together, and just creates a .fit file when it's started, and writes GPS data to the file until it's stopped.

Pulling it all together

So with all those pieces in place, I've made it so that as soon as the main service detects that network is available (via the network service), it attempts to upload all FIT files which are in the flash filesystem. If the upload is successful, it deletes the local copy.

As soon as the GPS gets a lock, the logging service is started, which starts recording GPS data. When the device is powered off, the file is finalised and stored to flash, ready for upload next time the network gets connected.

With a dead simple golang HTTP server, I can receive the files and save them to my computer:

package main

import (
        "fmt"
        "io/ioutil"
        "net/http"
        "os"
        "path/filepath"
)

func run() error {

        fmt.Println("Listening...")

        http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
                err := r.ParseForm()
                if err != nil {
                        fmt.Println("failed to parse form")
                        http.Error(w, fmt.Sprint(err), http.StatusInternalServerError)
                        return
                }

                v := r.Form.Get("filename")
                if v != "" {
                        b, err := ioutil.ReadAll(r.Body)
                        if err != nil {
                                fmt.Println("failed to read body")
                                http.Error(w, fmt.Sprint(err), http.StatusInternalServerError)
                        }

                        fname := filepath.Clean(filepath.Join(".", v))

                        fmt.Println("Writing to", fname)

                        f, err := os.Create(fname)
                        if err != nil {
                                fmt.Println(err)
                                http.Error(w, fmt.Sprint(err), http.StatusInternalServerError)
                                return
                        }
                        defer f.Close()

                        _, err = f.Write(b)
                        if err != nil {
                                fmt.Println(err)
                                http.Error(w, fmt.Sprint(err), http.StatusInternalServerError)
                                return
                        }

                        fmt.Println("Done.", fname)
                }
        })

        return http.ListenAndServe(":8080", nil)
}

func main() {
        err := run()
        if err != nil {
                fmt.Println(err)
                os.Exit(1)
        }
}

And my favourite part is that the generated FIT files are consumed just fine by the Garmin Connect website :-D

Here's a little walk around a local green:

Complete with a couple of stops to avoid cows: