Project Logs

Collapse

• Containers are cool now

  Stephen B • 12/27/2020 at 16:18 • 0 comments

  I wanted to upgrade my server to Ubuntu 20.04, so I've redone the packaging of the server side of this project as a docker compose script.  Using 250MB of containers to do the job of 25k of scripts isn't something I'm proud of, but I've got a few other containers running now so it fits into a flow.

  I've also bumped the dependencies to ones without vulnerabilities in (Pillow), moved from Python 3.6 to Python 3.8, unified on a single XML library (lxml), and made the logging less crap.

• As with comedy, timing is everything

  Stephen B • 01/27/2017 at 10:25 • 1 comment

  Actually I'm talking about time zones.I finished this project in early March 2016, handed the gift over and hoped that I would not be supporting a one-off gift for the rest of my life. Then the clocks in the UK moved forward an hour.

  Shit.

  The micropython board does not understand time zones, and without retrieving it across the country, never will. So the server must, and translate. In a problem mirroring Unicode handling in Python 2, I solved this by marking each time source with a timezone using the Python library pytz, and then translating time data between time zones.

  This seemed simple but no source explicitly records time zone information - I'm using:

  • An XML API
  • An HTML page I'm scraping with beautiful soup
  • Stored data in XML (regretted that pretty quick)
  • Stored data in JSON

  And all need to have time zone information added back. 'Ware time zones!

  ---

  If that wasn't enough, around 31st December a forgotten TODO in my code came back to bite me. The tide information I scrape doesn't actually include the year. "That's alright," I said, "I'll assume it's the current year". This is not true of course at the end of the year where all future tides are in the next year. In theory this would sort itself out on New Year's Day but I decided to fix it with a year increment when Dec rolls to Jan on the page.

  I frantically fixed this before remembering the clock has been turned off for two months while my step mother is on a cruise. Ho ho ho.

• Power

  Stephen B • 03/27/2016 at 21:58 • 0 comments

  Those who have stuggled to reconcile that the server generates the whole image and the display is dumb and provides no input with the fact that there's battery information from the display should pat themselves on the back now.

  Before the display requests the EPD data it sends a GET request with query parameters and one of them is the battery charge information. I did think about uploading its log but in my experience that's great until the thing that breaks is the log-uploading and then you're still back to reasoning if there's network problems or something else.

  Sleep

  While we're on power I've not mentioned sleep currents. The EPD claims less than 0.1uA. It must be true as the Fluke meter I used has a noise floor about that level. The WiPy claims 7uA. This is unfortunately not true. WiPy + expansion board + SD card gets me 700uA. The SD card can't be powered off - I was originally going to use that for logging but I can save 150uA by losing that. The resistor network measuring the battery voltage is using 40uA which I could have removed or switched if I was feeling keen. The actual charging IC appears to be using 200uA, still leaving (roughly) 350uA left.

  This was pretty disappointing, and I've not seen any satisfactory response from the WiPy manufacturers to the posts I found on their forum about it. Maybe there's a new hardware release that fixes this. Or maybe they copied off the sleep power stats of the TI SoC and didn't consider what their board actually uses. Who knows?

  In the end I solved it by using a bigger battery and deciding that charging it more often isn't the end of the world. This did have the advantage that the bigger battery coped better with the startup of the WiPy.

• Timing

  Stephen B • 03/27/2016 at 21:43 • 0 comments

  The WiPy has an RTC, but it needs initialising on boot (and wakeup from deep sleep too) even without power loss, don't know if that's a limitation of the micro or the RTC or Python or what. So we need a time. I put this problem on the TODO pile and started writing the network code, more specifically parsing HTTP headers

  HTTP/1.1 200 OK
  Date: Sun, 27 Mar 2016 21:36:31 GMT
  Server: Apache/2.4.7 (Ubuntu)
  Last-Modified: Sun, 27 Mar 2016 21:30:01 GMT
  ETag: "94b-52f0e7fb04995"
  Accept-Ranges: bytes
  Content-Length: 2379
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: image/png

  Oh hey, a time! Let's assume it's accurate and use that!

  These headers are from Chrome, I worked on HTTP 1.0 rather than take the risk I'd upgrade Apache or tinker with it in the future and find it returning chunked data for some reason which I haven't written support for. I do support keep-alives though, to save valuable milliseconds awake and sending WiFi.

• Costs

  Stephen B • 03/27/2016 at 21:29 • 0 comments

  For those interested in making their own, some rough costs:

  • The EPD and its controller were just under £60 including delivery from the US
  • The picture frame was £45
  • The WiPy and its expansion was about €36 plus postage in the kickstarter, I don't know how much they are now

  If I were to do this again, there's obviously money to be saved by porting the Python to C and using a cheaper micro. Although to be honest, unless I was making a lot of them, I wouldn't bother. Being able to charge a Li-ion battery off USB was quite useful from the expansion board certainly.

  A cheaper picture frame would help but going to a proper framing shop meant they let me take a bunch of frames apart until I found one that fitted and so that was worth the cost.

  The EPD is just expensive. It would be cheaper to gut a Kindle and use that. I looked at it briefly, looked at the -20V and the +22V and the other weird voltages and thought "nah".

  Feel free to leave comments here on how I could have done this with an ESP8622. I have one on the desk to play with next - I won't even disagree with you.

• Data sources

  Stephen B • 03/27/2016 at 20:55 • 0 comments

  This uses UK government data sources for the tides (UK Government Hydrographic Office) and for the weather (Met Office). The server part of the tide clock parses the XML from the Met Office from the land forecasts feed and from the sea observations feed. There's not many sea observation points. Really! There's about ten that actually have information about the sea itself (as opposed to visibility and the like). I wanted sea temperature. The documentation is pretty good, just needs a free API key to access.

  By contrast UKHO have an EasyTide application which is not designed to be machine parseable, but it puts out a "printer friendly" format that removes enough cruft to be attacked with BeautifulSoup, a Python HTML parser. It also only puts out times in GMT and not BST despite saying it has DST support.

• Communicating with an EPD

  Stephen B • 03/27/2016 at 17:27 • 0 comments

  I was pretty excited when I found the EPD module for sale on digikey. It does all the generation of the weird voltages for the display and allows communication by SPI. I can do SPI! The sleep current is in nano-amps when it's off as from what I can tell the EN pin must be to a low current FET that just drops literally everything on the board.

  Talking to the board was really easy in Python, I could use the REPL (read-evaluate-print-loop, the interactive shell as I think of it) to initialise the SPI module in different ways and send commands, and see whether I'm talking to it correctly. In C this is a slow process, but in Python without needing to recompile code I could work out the many many bugs I tend to bring to SPI-protocols in an hour.

  Then the display didn't work at all - lots of panic ensued but happily it eventually turned out I had been too delicate with the edge connector and it needed better seating before the module would talk to the display itself.

  The module is basically another micro with its own flash that can store images and update them and cycle between them with few commands to it. It doesn't entirely work in its specification (e.g. sending image data can *not* be done with a 50ns gap between packets like the datasheet says, it's more like 500us), but with some slack it does work.

  Images

  The image format used by the display is deliberately simple. There's a 16 byte header, and then the raw image data, one bit per pixel. So for the 400×300 display, that's 15000 image bytes, and 16 header bytes for a file size of 15016.

  I haven't been able to calculate a checksum that matches the checksums that the board calculate, so I've not been able to confirm uploads to the board are correct. Comms can in theory be up to a few megabit but I've left it at 100kHz as the wiring isn't shielded.

• When is a log not a log?

  Stephen B • 03/27/2016 at 17:15 • 0 comments

  I'm writing snippets about the development of the tide clock here, but they're not in real-time, as for nearly the entire duration of the development, I wasn't sure this would actually work properly. The display doesn't always update properly but is doing this infrequently (and certainly not when I refresh the board in a loop trying to trigger the behaviour). The server-side code is pretty hacky, and I'm not really happy with the fly-wires inside the picture frame either.

  The original plan was to laser-etch the map onto plywood and mount the PCB on the back of the wood, but due to some boring life cock-ups, this didn't pan out, so the mounting onto card was more of a last-ditch plan.

View all 8 project logs