A pen with optical text recognition that reads text aloud ... potentially helping the visually impaired

Similar projects worth following
In summary, this is an ambitious project aiming to initially perform individual character recognition with a 30x30 mouse sensor camera, and ultimately reading text aloud.

As most would know, mouse sensors have fast but low resolution cameras used to detect movement. Importantly, mouse sensors are small and much easier to interface with than high resolution cameras. By using a low resolution, the potential to interpret and hence perform optical text recognition for individual characters is theoretically possible. However, till now, I have only seen basic OCR done with numbers on PICs. OCR is not easy, especially with microcontrollers. Hopefully -- yes -- hopefully a Teensy 3.1 will be able to do it.


  1. Performing OCR with the mouse sensor and detecting rotation of the text (text cannot always be expected to be upright/vertical).
  2. Performing OCR at a decent frame rate to allow reading of text
  3. Shrinking the hardware down to "pen" sized

Obviously, there's a lot more than that. Consider this the first stage of the project.

  • Project stopped

    Brian Chen12/09/2015 at 03:39 0 comments

    Unfortunately the project stopped a while ago, since it was simply too difficult to get reliable data from the mouse sensor. The sensor did not seem to be able to see text even with different lighting configurations. Should anyone wish to do something similar to this project, I suggest they use an actual camera sensor.

    Recently I've discovered that pen-like products already exist in the market that perform OCR. So the foundations are already there. What in fact needs to be implemented now is some way for the pen to communicate to the user physically. I've had recommendations from friends to use some sort of mechanical system on the pen that creates braille physically on the pen. After research, I have discovered that braille characters can pretty much be represented by a 3x2 matrix. Thus, only 6 physically on/off pins need to be mechanically made. Nevertheless, this is still a rather daunting task and I'm sure their are flaws to the idea.

    For now I'll be working on my new project: the ballbot.

  • Positioning the mouse sensor

    Brian Chen06/26/2015 at 02:44 0 comments

    A typical computer mouse has the sensor elevated at just the right height. For the ADNS-9800, this is approximately, this is approximately 2.4mm. HOWEVER this is not a concern for our application as we aren't detecting the small particles of the surface. Instead, it's important to position the sensor at exactly the right distance so that individual letters are captured on the camera. As a result, the "pen" may look less like a normal "pen" but instead a tubular device that needs to be in contact with the paper.

  • Working with the ADNS-9800

    Brian Chen06/09/2015 at 07:06 0 comments

    So I started the project today, hoping to get the camera showing some images of text on the computer. I succeeded ... partly. The camera seems to struggle with differentiating between anything but extremely bright and dark objects. In the end, I was only able to recognise bright sources of light (such as the ceiling light) as pictured below.

    (instead of writing a proper application for showing shades, I decided to just send different ASCII characters over serial).

    I've also tried with and without the lens, and it seems that removing the lens gives better results.

View all 3 project logs

Enjoy this project?



Matthew Chen wrote 06/09/2015 at 09:35 point

How to you get the processing speeds to process the data though.

  Are you sure? yes | no

Brian Chen wrote 06/10/2015 at 10:34 point

As in processing power to perform the OCR? Good question. This is something something that can really only be found out once I get to that stage. Using very simple methods for OCR would that would be least computationally expensive comes to mind, though this may be a tradeoff for text recognition reliability. Even that might not be possible

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates