Close

Writing Software to Track Eye Movements

A project log for HeadsUp, a Low-Cost Device to Diagnose Concussions

Diagnose potential traumatic brain injuries on the sidelines quickly, cheaply, and conveniently using a 30-second eye test.

mihir-garimellaMihir Garimella 07/11/2016 at 06:060 Comments

Now that we had some hardware to work with, we needed to write software to track a user's eyes while we showed them a visual pattern.

The second half of this—displaying a pattern on the LED's attached to the Spark Core—was the easy part, so we did that first. We created a cloud-connected Spark.function called "start" that would turn the bottom, left, top, and right LED's on for six seconds each (in that order). Then, whenever we needed to start displaying the pattern, we used cURL to call the function over the Spark Cloud. This was all really simple, and accomplished in ~50 lines of code.

The next step was actually tracking a user's eye movements. We brainstormed methods and prototyped them in OpenCV. For example, an early implementation found the x-coordinate with maximum contrast (going from the white part of the eye to the iris/pupil and back to the white part) to locate the pupil horizontally, and then found the darkest cluster of pixels along that x-coordinate to find the center of the pupil vertically. Here's a photo after the program found the horizontal position of the pupil:

We also tried a few open-source implementations, like this one. However, we found that none of these methods were robust; one particularly difficult problem was when the user was looking almost straight down at the LED on the bottom of the device; their eyelid would be covering most of their eye, so only a small, non-circular portion of the iris would be visible.

We kept thinking and ultimately came up with the algorithm that we ended up demoing at PennApps. We first passed each camera image through one of OpenCV's Haar cascade classifiers to isolate the eye and remove the rest of the face. Then, along the vertical line at the horizontal center of the eye image, we found the number of white pixels before and after the darkest cluster of pixels, and we repeated this procedure along the horizontal line at the vertical center. Then, we compared the number of white pixels on either side of the dark cluster for both directions to determine the direction of the user's gaze. For example, if the amount of white pixels on the right side of the iris was far smaller than the number of white pixels on the left side of the iris, then the user was most likely looking to the right.

We implemented the final algorithm in C++. We couldn't run it in real-time because we weren't able to find a PS3Eye driver for Mac that allowed us to read the cameras from OpenCV, so used an open-source viewer app to display the PS3Eye camera feeds on screen, used QuickTime to make a screen recording, and passed the resulting movie through our eye tracking program. Instructions for testing this out (along with a recording that we made during testing) can be found in the readme of our Github repository.

(In our next prototype, we definitely plan to modify our approach to be able to perform the eye tracking and diagnosis in real-time).

Discussions