13 days ago •
User Interface Design
Visualization and user interface design is a hobby that I really enjoy, and I've been lucky enough to have friends and colleagues who are usability and visualization folks that I've been able to periodically soak information up from. The open source science tricorders present a really interesting UI problem -- where most mobile devices have at most a few different sensing modalities, here we have 12 different physical sensors, each of which measures between 1 and 3 different things! What's more, while some of these measurements are single values (like temperature, say 24C), others are vector values (like the three separate 3-axis vectors coming from the MPU9150 -- one vector for acceleration, rotation, and magnetic field strength, giving nine values total, from a single sensor). In the extreme case, the spectrometer returns a vector of 256 values. That's a lot of data! I'm really not aware of any other device that comes close to having so many different kinds of sensing data pouring into it constantly, and while this is very exciting, it's also challenging -- we want folks to intuitively browse and navigate through that data very quickly.
I've spent the past few weeks researching design concepts for user interfaces by browsing popular design websites, and talking to some friends, and I think I've settled on a very intuitive, attractive, and useful interface design. I'll confess that I've been working harder, not smarter about this in recent years -- I'd try and design very complicated things that looked like how other people had done mobile devices and visualization (like android phones), which is a fantastic amount of work for a single human being. Starting this Arducorder mini project almost 4 months ago, and keeping it attractive and capable but tractable has finally helped me work smarter about this, and figure out a lightweight, usable and intuitive interface concept.
I like the idea of live tiles. When done well, it reminds me of Danny Hillis's idea (30 years ago) about maximizing the amount of computing silicon active on a processor at any one time, but attractively applied to data visualization. Instead of having icons that just take up space and give you a name, here you have tiles that display a live updating value. This is powerful from a usability perspective -- instead of having to enter each application to obtain this information (which can consume a lot of time), the most important information is available at a glance from many applications. While this has largely been applied to communication (e.g. messaging), web (weather/news), and I'm sure sensing applications, here we're going to construct an interface made almost entirely of sensor data.
The really interesting aspect (if you're a data abstraction nerd, like me) is that if we consider only this special case -- visualizing sensor data through tiles -- then we can formulate the software engineering aspect of this very elegantly. If each tile has a sensor data buffer behind it, then we can browse it's most recent data on the top of the tile, then activate the tile to display additional (historical) data and different visualizations. If we abstract the sensor data buffer into a few different types, like continuous streaming data (from something like a temperature sensor or an accelerometer) and discretely sampled data (from something like a lightning sensor, that only activates when an event occurs), then we can get rid of the idea of an application. The design concept becomes browsing sensor data buffers first through a high-level tile interface, and then activating an individual buffer and using a suite of generic visualization tools to better explore it.
All of the software engineering aside, what the user sees is very attractive -- tiles with live data that they can browse through very quickly with...
Read more »
a month ago •
A quick update with lots of progress designing the spectrometer (and thermal camera) board, attaching the radiation sensor board, and beginning to work on the low-level routines for the user interface!
Fast and Efficient Bitmap Rendering
Now that much of the hardware is in good shape (or being fabbed), I've been working on the low-level software and APIs that would support building an intuitive and attractive user interface.
A bit of background -- from my hobby of indie game development in undergrad and participating in the Ludum Dare 48-hour game competitions (a really fun experience!), I've picked up that in a desktop system (and some embedded systems), a graphical user interface is often functionally implemented as a large collection of bitmaps stacked atop each other with transparent layers. This has the benefit of being able to make large changes to the user interface through editing configuration files that describe which bitmap corresponds to which background or widget or font (instead of recompiling code), and also allows the user interface artist to easily use whatever tools they're most comfortable illustrating in to rapidly iterate without having to use middleware, or (even worse) be constrained to building an interface entirely out of low-level graphics primitives like squares, ellipses, and triangles.
This is the approach that I'd initially planned on using, but I'd discovered about a month ago when running a display verification test that the ChipKit SD card library has a read speed of around 100k/sec, which puts the time to load and display a bitmap on the order of about one second -- far too slow for an interface. There are faster libraries like sdfat, but these required more than a solid evening of porting, and at some point you have to put these on the wishlist, accept the constraints, and exploit the benefits of your existing system while architecting it in such a way that you could drop more advanced functionality in later with a minimum of effort.
While loading bitmaps from an SD card is currently slow, the PIC32MX processor that the Chipkit Max32 uses does have 512k of flash memory that (in addition to code) can store bitmaps that can be loaded very quickly. A full screen (128x128) 16-bit (2bpp) bitmap takes 32k of memory, so we can't fit very many in memory, but if we manage our resources and store things efficiently, we should have plenty of space for backgrounds and widgets and icons in ~128k or so.
One way of compressing bitmaps is to reduce the colour depth -- if we can reduce an image from 16-bit colour (64k colours) to 8-bit (256 colours) or 4-bit colour (16 colours), then we can store the image in half to a quarter of the space. The trick is selecting a smaller colour palette so that the final "posterized" image still looks very good.
There are automated algorithms to do this, but they often give non-ideal results, and I want this thing to look /good/. I've been looking at user interface concept art the last few days to get ideas, and I'd had the idea of having a main menu that showcases a space theme by having a scrolling background under an icon-based main menu with multiple pages of icons, similar to Android. I grabbed a copy of NASA's amazing public "earth at night" photograph, downsampled it to 256x128, and used manual posterizing tool I quickly put together in Processing to reduce the colour depth to 4-bit, and store the image as a static array in flash. Despite being 256x128, the image only uses 16k of flash, and the manual conversion (above) is so good compared to an automated algorithm that I would never have known it was poserized if I didn't do it myself! It also scrolls back and forth at about 50fps, which looks really impressive. While I'm still deciding the final user interface concept, it was a good test, and a useful tool to compress the graphic assets into flash so that they're...
Read more »
a month ago •
A quick update, with some of the high-risk high-gain aspects of the project!
CC3000 WiFi Module and ChipKit Library Port
It's great to get out of your comfort zone and learn new skills or expand your familiarity with different aspects of design. For me, in addition to the capacitive touch wheel, I decided to include the popular CC3000 WiFi module from Texas Instruments. I think connectivity fits a lot of the use cases for a science tricorder-like device, from kids using social media to share their sensing and measurements to engage themselves and their friends, to nerdy scientists that spend their entire days in the lab (like me) who would like to push all their raw data to somewhere like data.sparkfun.com for sharing, advanced visualization, or later analysis. Personally, a neat visualization experiment I'd like to try is to sit the Arducorder Mini on my desk for an afternoon when we're expecting a storm, and visualize the storm coming in through data -- the decreasing temperature and pressure, increasing humidity, reducing distance-to-lighting, and it'd be neat to see if any x-rays from close lightning strikes are detectable and coincide with the lightning detection.
This is my first use of a WiFi module in a project, and so my concerns in terms of hardware were routing the antenna traces correctly, and being able to successfully interface with the module using an open library. There are a number of open reference designs for the CC3000 antenna circuit, and space being at a premium I ended up using the same layout as the folks who designed the open Spark Core (thanks Spark team!). In terms of open ilbraries, Adafruit released their CC3000 Library, which was recently ported to the Due, which would give me a trail of github commits to look through when porting to the ChipKit MAX32-based system that the Arducorder Mini is based upon. It all looked good, so I incorporated the CC3000 module into the Arducorder Mini motherboard.
To make a bit of a long story short, I ran into a problem similar to the issue with the capacitive touch sensor requiring the ChipKit I2C library to be updated from the Arduino 0023 libraries to the new Arduino 1.x (post-Due) libraries, to make use of the additional features that have been added in the last few years. With the capacitive touch sensor, it was fairly easy to add the repeated stop condition support from the newer Arduino 1.x I2C/Two-Wire libraries, but the Adafruit CC3000 library is based on the much larger Arduino 1.x ethernet library, so there was a good deal of effort involved -- porting the Adafruit CC3000 library from the AVR/Due to the PIC32 from the top, while simultaneously updating the ChipKit Arduino peripheral libraries from the bottom.
Fully updating the ChipKit libraries to Arduino 1.x would be a full-time task for someone familiar with their inner workings for a good week or two, so it's well beyond the scope of this project -- but after a solid weekend of work I was able to successfully port enough of the Arduino 1.x ethernet library (and its dependencies) to then port the Adafruit CC3000 library to the ChipKit! The picture above shows success, and the Arducorder Mini successfully fetching its first webpage and displaying it over the serial console. Great stuff!
In the end, this means that in order to compile the Arducorer Mini firmware, I'll also have to maintain an updated version of the MPIDE on the github repository. A side effect of this will be that Chipkit folks should be able to enjoy the benefit of some (partial) Arduino 1.x compatibility.
Open Mini Spectrometer
In addition to the WiFi module, the other high-risk high-gain aspect of the Arducorder Mini is creating a new version of the open mini spectrometer suitable for one of the low-profile modular sensor boards. I really needed to build (and not code) last weekend, and have been boring...
Read more »