So I managed to write a bash script that

• launches livestreamer in the background, which streams to ffmpeg, which writes images captured from the stream to disk
• scans for those files and finds the newest one to compare with a reference image
• deletes old images
• kills the background livestreamer when it exits (this was hard-ish because of my low google-fu)

Here it is:

#!/bin/bash

threshold="0.1"

# http://stackoverflow.com/questions/18279540/bash-shutdown-hook-or-kill-all-background-processes-when-main-process-is-kille
trap 'jobs -p | xargs --no-run-if-empty kill' EXIT

livestreamer --player "ffmpeg -i" --player-args "{filename} -vf fps=0.1 out%04d.png" https://www.ustream.tv/channel/17074538 worst &

while true; do
    # find newest file and compare
    FILE=$(ls -t out*.png | head -n 1)
    { output=$(compare -metric MSE -fuzz 20% $FILE references/novideo.png tmp.jpg 2>&1 1>&3-) ;} 3>&1
    # extract and compare, output binary value indicating if HDEV stream is available or not
    value=$(echo $output | cut -d "(" -f2 | cut -d ")" -f1)
    echo $value'>'$threshold | bc -l 

    # remove old files
    # http://stackoverflow.com/questions/25785/delete-all-but-the-most-recent-x-files-in-bash
    (ls -t out*.png | head -n 2;ls out*.png)|sort|uniq -u | xargs --no-run-if-empty rm

    # check if livestreamer is still running and restart if required
    # TBD ...

    sleep 5

done

• if the livestreamer process dies for some reason, I don't have a way to detect that and restart it (yet)
• temporary images and the compare result image should be written to RAM, because I want this to run on a raspberry pi. Writing this many images to the SD card will wear it out sooner than later.
• I'd like to publish this as a mqtt topic, but available to the public