Close
0%
0%

TextEye: Raspberry Pi (Zero) Mobile Textreader

A mobile text scanner/reader for people with severe visual impairments

Similar projects worth following
The idea behind this project is to create a mobile device that can be pointed at signs, menu cards, books, magazines etc. and on the press of a button read them to you.

Using the Raspberry Pi, preferably the compact Pi Zero, a camera module, audio output and a rechargeable battery in a yet-to-be-designed case, it should be possible to achieve this by a combination of software for taking images, OCR processing and text-to-speech conversion.

Later on, the unit could be combined with a fixed stand in order to use it as a book or magazine scanner. This could also be connected to a bigger display for showing text with different zoom modes.

The Idea

After watching Ladyada's intro video to the Pi Zero contest (see it here if you haven't already), I was thinking about what might be an interesting project with a Raspberry Pi Zero.

Somehow the idea for a mobile text reader for visually impaired people popped up in my head. While I currently don't have much contact with such people, I actually got to know one blind guy very well several years ago. He was a friend and colleague of my father, and we often visited him and his family. He was living a full life and clearly enjoying every bit even in spite of his blindness, so that was a pretty positive impression I got.

Years later, when I started to study at the University of Karlsruhe, we had a group of blind people studying computer sciences and other topics along with the rest of the students. At that time it was a prototype program that incorporated new hard- and software, like mobile braille terminal laptops, OCR etc. and special oral exams into a new offering for enabling people with severe visual impairments to study the same thinks as everyone else.

Due to that, my first idea was to produce something similar - a small mobile unit with a camera and integrated OCR and a mini braille display output. Looking at the braille display technology though, I realised that a usable electro-mechanical device like that would need more time than I would have if I wanted to enter this project into the Hackaday / Adafruit Pi Zero contest.

So instead of adding a braille display, I chose to use text-to-speech conversion and audio output instead.


Options for Expansions or Variations

Thinking about different components and build variations, there are several options to expand the basic unit or turn the construction into a different variation, depending on individual needs. Here are some of the possibilities I've come up with so far:

  • a fixed stand with a slide-in mechanic that the mobile text reader can be connected to - allowing easier battery reloading, easier book scanning, and adding the option of hooking it up to a TV or monitor for zoomed text viewing
  • expanding the software to incorporate a text library (that partially might have been created using the OCR software) with an audio interface, turning it into an audio text player
  • adding a braille display (at least for a non-mobile usage) for text output, with the option of audio output of displayed lines for learning
  • adding an LCD or OLED display for turning it into a mobile electronic magnifying glass

Hardware components

The necessary hardware components for the basic device should be the following:

  • Raspberry Pi (Zero or other)
  • Pi Camera module or USB webcam
  • for the Pi Zero: DAC board for quality audio output (optional for the other models as those have audio output included)
  • rechargeable battery
  • recharging circuitry for the battery
  • speaker for audio output (optional)
  • ultrasonic or IR distance sensor (optional)
  • buttons for scan activation, volume control, mode switching etc. (depending on intended software and hardware combination)
  • additional SD card reader/writer breakout board for "external" image and text storage
  • optional vibration motor for feedback from distance measurements

Software components

The basic device should (at least) be able to do the following things:

  • taking images and storing them onto an SD card
  • running OCR processing in order to convert the images into text files
  • creating audio output from the text files using text-to-speech conversion

Thankfully, those main functions have already been implemented by several people in different projects.

Camera function

Greg Holloway has documented his SnapPicam project on the Adafruit Learning system: https://learn.adafruit.com/snappicam-raspberry-pi-camera

This project already includes the first block of functionality I need here - the ability to take pictures on a button-press and store them as an image file.

There are other documentations available elsewhere on how to install and use the Raspberry Pi camera. Concerning energy usage, I'd prefer using the...

Read more »

  • 1 × Raspberry Pi The current prototype uses a Pi model 2. The plan for the intended first non-prototype version is to use a Pi Zero. It should also work with other Pi models, but you may need to adapt the case design.
  • 1 × Raspberry Pi Camera module (alternative: USB webcam) Thanks to the new version of the Pi Zero, a Pi camera module can be used for taking pictures. Alternatively, you can use a USB webcam (which you can dismantle a little in order to make it smaller).
  • 1 × Raspberry Pi Zero camera connection cable This is only needed if you use the new Pi Zero and Pi camera module for the build. For other Pi models, you can use the cables which are included with the Pi camera module.
  • 1 × SD card One 8GB SD card (in micro SD format) is needed for the operating system and applications. A second SD card can be used in combination with an additional SD card reader in order to be able to more easily transfer the image and text files to another computer.
  • 1 × USB to TTL serial cable (alternative: network adapter and cable or WiFi adapter) This is mostly needed for connecting to the Pi during prototyping and further development, and for installing software. Once the build is finished, it's only needed for updates - unless you add a WiFi adapter and connect to the Pi wirelessly..

View all 14 components

  • A light change...

    Markus Dieterle10/09/2016 at 17:37 0 comments

    High time for another log update...

    I'm currently on vacation, but I took most of the prototype parts and a computer with me as I hoped to get some additional work done (only to find out that the vacation schedule was a little too dense for that - well you need to get some relaxing done during holidays I guess :) ).

    Breadboard prototype - current iteration

    First of all, let's start with my latest Fritzing diagram for the breadboard wiring of the TextEye prototype:

    Since I cannot currently find Fritzing parts for the newer version of the Raspberry Pi Zero or for any Raspberry Pi camera module, I took the older Pi Zero and a similar looking part and connected them with some direct line connections (although there is no actual connector on the Pi Zero right now), At least it is a somewhat realistic depiction of the actual prototype setup.

    Now this looks pretty usable, but the physical prototype ended up being a little different:

    If you compare the physical prototype and the Fritzing diagram, you'll notice the following differences:

    • the physical prototype uses a Raspberry Pi Model 3 instead of the Pi Zero as the additional connectors simplify the development (updates, testing etc.)
    • the PowerBoost 1000C is not yet connected to the prototype setup - mainly because I did not get around to the additional soldering before my vacation
    • a TTL to USB serial cable (https://www.adafruit.com/products/954) was added for a direct serial connection to the Raspberry Pi
    • since I did not bring enough breadboard jumper cables with me, one of the pushbuttons is currently not connected to ground (not a big problem right now)
    • last not least: the NeoPixel ring has disappeared and was replaced by the "Bright Pi" board:

    The Bright Pi board uses simple white and IR leds instead of the RGBW leds of the NeoPixel ring or strip: https://www.pi-supply.com/product/bright-pi-bright-white-ir-camera-light-raspberry-pi/

    Software (again)...

    The main reason for using this "fallback" option for the lighting lies within the software development.

    Since I'm using the C/C++ programming language for software development, I had already decided to use the "WiringPi" library for the GPIO access. When I looked at how to use the NeoPixel elements however, I found that the current option for this would be to use another library. Even though the other library would not access all GPIO pins, I could not be sure that the functions of both libraries would work alongside each other.

    My programming experience related to hardware access tells me that a scenario like this, with functions from different libraries probably "fighting" over control of the same GPIO channels, is a potential recipe for disaster - unless you wrote both libaries yourself and know how to properly tweak them and make them work together.

    From my research, I know that NeoPixel control mainly needs properly quick PWM access. With some additional time, it should be possible to use WiringPi to create a usable NeoPixel control library in C/C++. In order to get closer to a working prototype sooner, I chose to use the Bright Pi board instead.

    The Bright Pi board is actually an I2C device with a simple 4-pin connection. Once I2C has been enabled on the Raspberry Pi through the advanced options in "raspi-config", you can pretty much follow the code examples on the Bright Pi web pages and combine them witn the WiringPi I2C examples in order to control the lights on the board from your C/C++ code.

    While I have not finished the software yet (maybe I'm complicating things more then necessary...) I have got around to writing a quick and dirty test program today. I hope to get around to properly testing the hardware setup with this during the next few days.

    Video todo

    Due to my vacation (and a distracting bad cold) I'm still not as far along as I would like to be, but it's definitely no standstill.

    For the Hackaday Prize, I still need to produce a short video showing of the project and a working prototype. Since the deadline for the main prize is October...

    Read more »

  • Changes and Upgrades

    Markus Dieterle08/12/2016 at 15:02 0 comments

    It's high time for an update...

    Since I did not add another log entry for quite some time now, this one is going to be somewhat longer. Let's start with the TL;DR version first:

    • breadboard prototype is (almost) finished
    • Raspbian and open source base software were reinstalled after some technical difficulties
    • the TextEye software is undergoing a change from the initial C++ design to a more modular standard C design
    • components for version 1 hardware prototype design were changed:
      • Raspberry Pi camera module replaces USB webcam
      • Raspberry Pi Zero was upgraded to version 1.3 (with integrated camera connector)
      • NeoPixel ring replaces NeoPixel strip
      • MAX98357A breakout board replaces the previous combination of DAC and amplifier breakout boards
    • Raspberry Pi model 3 board was purchased for more direkt Pi development
    • new digital camera for documentation and upcoming videos
    • added 3D printing capability for custom case design

    If you'd like to know more, keep on reading...

    Breadboard prototype

    The hardware changes described above have slightly affected the breadboard prototype, but not too much:

    I'm plannig to update the Fritzing design after finishing the breadboard prototype, but that requires creating some new parts in Fritzing as I'd like to add the Pi camera module and connector cable along with the new Pi Zero v 1.3 board (for which there does not yet seem to be a ready made part in Fritzing).

    Software reinstallation

    In order to more easily transfer code and image files between the Raspberry Pi and my development system (and more easily create backups), I registered a USB stick and created a new mount point on the Pi. Mounting the stick, reading and writing to it worked fine so I decided to add the device to the device table and configure it for automatic mounting on startup.

    The result of this was that the Raspberry Pi did not complete the boot process any more. No matter if had the USB stick plugged into the Pi or not, it just hang up during boot. I could not connect to the Pi any more, wether I tried the WiFi connection or the USB serial TTL cable connection.

    So I went ahead and flashed the SD card with a new, updated Raspbian Jessie light image. After basic configuration, I reinstalled the software: GraphicsMagick, Tesseract OCR, Festival and WiringPi. As a final step, I copied my development folder with all the code, test images etc. back to the Pi SD card (thankfully I'd taken care to regularly save the code etc. before this misshap).

    Another lesson learned... :)

    TextEye software redesign

    After finding some time for working on the code again, I had gained sufficient distance to it in order to see that it was getting to complicated and not modular enough.

    With the experience from my tests and necessary future changes to different parts of the workflow in mind, I knew I needed a different code design that would allow easier swapping of the code within the individual workflow steps.

    So I've started to do a rewrite using the standard C language, with different header and code files for camera functions, image processing, text recognition and speech output along with generic headers and the main file.


    Hardware changes

    Since I started this project, new hardware has become available that allows the TextEye to be smaller and a little more efficient.

    With the original Raspberry Pi Zero, I could not use the Pi camera modules as it did not have a camera connector port. The recently updated version 1.3 does have that, so I can switch from a USB webcam to the current 8 megapixel versions of the Pi camera or the NoIR camera module. That makes it more compact, more easy to power from the Pi and also upgrades the image resolution.

    The NeoPixel ring with 12 LEDs should work just as well as the NeoPixel sticks which I chose first with regard to brighten up the imaging area. It fits better into the overall design and also should slightly improve light distribution. I still have to test this option though. I've also gotten hold of the "Bright Pi" kit from Pi Supply as a possible...

    Read more »

  • Busy days, hardware frenzy and imaging insights

    Markus Dieterle06/03/2016 at 14:47 0 comments

    Hello and thanks a lot to everyone who's signed up to follow and like the TextEye project so far! You really keep me going even in the face of severe problems.

    I currently mostly work on improving the overall solution and finalizing the first proper version, but sadly don't have much to show so far. That should change over the next few weeks while I work on implementing a better time management so I can get more things done in the little time I currently have.

    So what's happening?

    Decisions on first version hardware

    Since I started the project, a lot of new hardware components have become available. Some of those are quite helpful for creating a really small and easy to carry mobile text reader solution, even more so than what I originally imagined.

    For example, the USB webcams work pretty well with the different versions of the Raspberry Pi, but you need to take them apart and also solder a shorter USB cable to them in order to get the size down. Thanks to the recent updates to the Pi camera modues and the Pi Zero - where the new version now comes with a camera connector - we can now use the new Pi Zero with a Pi camera module (normal or NoIR) and get a higher resolution than what most USB webcams can offer, all in a small, easy to fit package.

    While both camera options don't have the kind of shake reduction that you can get in modern digital photo cameras, it's a very convinient option.

    In combination with the Pi camera module, I'll try to add a small NeoPixel ring with RGBW LEDs instead of the NeoPixel stick I use for prototyping. This can fit around the camera lens, and it adds better white light for more brightness when taking pictures. It might not be necessary for the NoIR camera module, but I have not yet testet that (given that the OCR generally works better with greyscale images, the NoIR module might be a better choice as long as it produces enough contrast in the images).

    On the audio side, Adafruit recently came out with a new and nicely small mono amplifier breakout board (the MAX98357 I2S Class-D Mono Amp) that can be connected to a Raspberry Pi using I2S. While I was able to get the Pimoroni pHat DAC board which adds audio output capabilities to the Raspberry Pi Zero (and improved audio output for other Pi models), it took quite a long wait (about 2 1/2 to 3 months) because it was out of stock and took a while for new boards to be produced. From what I've seen from Adafruit - through a lot of their videos, including the "factory tours" Ladyada did on a few occasions - I'm confident that the new amplifier boards won't be unavailable for that long. And I like the smaller form factor - so I've decided to use the new board with the Pi Zero as part of the first version of the TextEye.

    So while I'm still prototyping with the Pi model 2, the first "usable" version should use the new Pi Zero, with camera module, NeoPixel ring, Adafruit MAX98357 I2S Class-D Mono Amp, mini metal speaker and USB-rechargeable battery. I've already ordered the new hardware modules and should get them soon.

    I thought that the Pi Zero might be a problem, as it has been hard to get in the last few months. But it actually arrived just today, after I ordered it directly from Pimoroni in the UK only a week ago (where the online store already stated that because of Pi Zero related shenanigans orders could only be handled with a 3 to 5 business day delay - and you can currently get only one Pi Zero per customer).

    As you can see, I went for the basic kit which included headers, USB adapters and a small case. I also added the camera cable which is needed in order to connect the Pi camera module (which I already have) to the new Pi Zero (the camera connection is a little smaller than on the other models).

    Imaging research and insights

    If you read through my earlier project logs, you know that while the different hardware and software components each work pretty well, using them for a mobile project like this takes them beyond the limits of what they were originally...

    Read more »

  • Breadboard noob :( (and DAC board update)

    Markus Dieterle05/17/2016 at 18:14 0 comments

    I'm slowly coming back to spending more time on this project. Progress could be faster - I still have to work on my time management skills, and maybe stop doing additional research at this point as it's taking too much time right now.

    Breadboard wiring errors

    Just as I was preparing the components for the hardware prototype wiring, I noticed some errors in my Fritzing designs - thankfully I noticed those before connecting and powering up everything. Apart from one missing GND connection line on one of the buttons and another missing crossover line for the 3.3V power line, I accidentally mixed up some power and ground connections - which could have been disastrous if I hadn't caught that before the fact.

    I have to admit that this is the first time I'm working with a standard plugin breadboard as well as the first time I'm working with the Fritzing software. Good thing the physical breadboard has some nice plus and minus signs for the default usage of the main lines along the long edges of the board...

    I fired up Fritzing again and corrected the errors. Here is the new version of the prototype layout using the Raspberry Pi model B+/2/3:

    And here is the same corrected sketch with the Raspberry Pi model Zero:

    In order to recreate this setup using real hardware, I need to do some basic soldering on a few components. Otherwise I won't be able to connect them to the board. Then again, once the setup is working, I'll have to do more soldering for the final prototype anyway.

    DAC option: PiMoroni pHat DAC board

    The incredible has happened a few days ago: I've finally received the PiMoroni pHat DAC board which I ordered quite a while ago (a litte more than 2 months if I remember correctly - I could look up the exact date):

    While I've already incorporated the Adafruit MCP4725D DAC breakout board into the current hardware design as an alternative DAC and audio output for the Raspberry Pi Zero, the pHat DAC board has the same form factor as the Pi Zero, and could possibly be a little easier to set up and use.

    The downside - as far as I can see it so far - is that the wiring connections to the other parts like the switches, resistors etc. can be a little tricky to do with the board connected to the Pi Zero trough the standard headers. I may have to add the additional wiring either to the back side of the Pi Zero or to the pHat DAC board on top, or come up with a kind of t-junction solution for the necessary pins. Or maybe using some "flatband" cable just for the few really necessary connections between the Pi and the DAC board, and single cable connections for the switches etc. is the better option - if anyone has worked on similar wiring options in the past, and can share some advice and experiences, please add them in the comments...

  • Fast Delivery

    Markus Dieterle04/30/2016 at 12:24 0 comments

    Just a short update:

    I only placed my order for the new Raspberry Pi Cameras on Thursday, and just today my neighbour gave me a small packet that was delivered sometime on Friday when I was absent - with just that delivery inside!

    So thanks to the great work from the people at EXP-Tech I've already got the latest Pi camera modules in my hands, along with two little PiMoroni camera mounts:

    Great stuff!

    I've bought quite a lot of stuff from EXP-Tech during the last few months, especially Adafruit components as they are one of the local distribution partners here in Germany. And while they don't offer all of the Adafruit parts, they are doing a great job nevertheless. Just a few weeks ago, they were really fast to send me a replacement board when I discovered that I had received the wrong one - I just wrote in and they promptly send me the right board the next day, before I managed to send back the wrong one to them.

    So if you are in Germany and need some electronics parts, especially Arduino-connected stuff and Raspberry Pi stuff, you should definitely check them out!

    I'll hook the cameras up to the Pi Model 2 and/or Model A+ in the upcoming days and take a buch of pictures with that. I'm really curious about the image quality...

  • Alive (and kicking... :) )

    Markus Dieterle04/29/2016 at 12:54 0 comments

    In the last few weeks my daytime job has kept me pretty buzy, mostly with a huge multi-month project that has just entered the "hot" phase.

    While this stops me from being able to work on the TextEye project as often as I'd like to, and will continue to do so at least until August this year, I haven't abandoned the TextEye project and will continue to work on it.

    Thanks to everyone who's been following this project so far - I hope you'll bear with me and keep coming back.

    For today's log I'll just do a short status update. Luckily, I had already planned a one week vacation period for the next week since January this year, so I should be able to give this project a much needed boost.

    So what's the status?

    Software status

    On the software side, I've created the basic structure for my "workflow" application and started to write a special C++ class for that which is designed to be flexible enough to easily switch out external and internal application components for creating, processing and OCR'ing the images etc. It's not finished though, and I'm currently thinking about creating a smaller, much simpler prototype app in order to speed things up a little.

    I also found an interesting, open source image processing app which is aimed at optimizing pictures for "digitizing" books and magazines. The software is called ScanTailor, and while it's not the newest piece of software, the last GitHub updates did not happen that long ago and it seems usable enough.

    On the downside though, it's essentially a desktop application which needs the visual checks to either acknowledge the automatically proposed optimiziations or do manual optimizations. With that, it can really optimize text images well enough to be usable for an OCR program. I have to take a closer look and maybe do some testing - if it has some command line options and could perform some of the needed image optimizations automatically, it may still be a usable option for preprocessing the camera images.

    Hardware status

    With the exception of the pHat DAC board from Pimoroni which hasn't surfaced yet, I've now got all necessary components for the prototype. The only other exception is a custom housing, but I plan to work on designing that after I have a working prototype.

    The hardware is not fully assembled yet, but I should be able to finish it in the next few days, at least far enough to do some initial testing.

    On the camera side, the Raspberry Pi foundation just released two new Pi camera modules on Monday this week: https://www.raspberrypi.org/blog/new-8-megapixel-camera-board-sale-25/

    While the Pi camera modules are designed to only work with the special camera connector slots on the Pi Model A+, B+, 2 and 3 - leaving out the Pi Zero - this is still an interesing option to use with the A+ (and maybe a 2 or 3 A+ version once one of those becomes available - which I'm hoping for). Consequently, I've ordered both the normal and the "NoIR" version just yesterday. I'm curious to see the kind of image quality that these newer sensors can provide. The modues come with integrated automatic white balance and the maximum resolution for stills is clearly better than what the external webcams can currently do (as there are no external 4K or higher resolution options for that at the moment).

    So once the basic hardware and software setup is working, I can see myself designing more than one reference platform design - one following my original design idea, with variations for the Pi Zero and the A+, another one using the Pi A+ and Pi camera, and possibly even a camera-less design that can be modified for being used with different digital point-and-shoot cameras. I've started working with OnShape a while ago while the service was still in beta, and I feel comfortable enough right now to consider creating my own designs for a custom case (or two or three... ;) ).

    It might take a while though...

    Moving on...

    I have to see what I can to in the next few days in order to move the project forward. It's not the only thing...

    Read more »

  • Prototype wiring preparations

    Markus Dieterle03/28/2016 at 16:37 0 comments

    Time for another project update...

    Breadboard prototype

    I did not do much coding over the Easter weekend, but instead took some time to get more familiar with Fritzing and think about how to wire up the different components. A few things are still missing, but here is the wiring for the breadboard prototype that I came up with so far:

    For the Raspberry Pi Zero, the prototype setup looks the same, just with the Pi Zero instead of the Model B(+)/2:

    The only thing still missing from the Fritzing sketches is essentially the USB webcam. I'm thinking about creating a custom part for Fritzing in order to complete the sketch in this direction.

    The NeoPixel strip in the upper right corner is just for additional lighting when taking a picture. It can be omitted or replaced by a bright white LED or even an external light source.

    The T-Cobbler is just used for a cleaner and easier connection between the GPIO pins on the Raspberry Pi and the rest of the parts. It can be considered to be part or the overall wiring.

    At the moment, the rest of the parts are as follows:

    • 1 Raspberry Pi (model Zero, A+, B+, 2 or 3)
    • 1 Adafruit PowerBoost 1000C charger board (the sketch shows a PowerBoost 500C as the library did not contain the 1000C model, but the basic connections are the same)
    • 1 rechargeable LiPo battery (a 2000mAh version in the sketch, but other compatible batteries can be used)
    • 1 slide switch for the PowerBoost on/off function (this may be replaced later by a different solution that allows the Pi to be shutdown properly)
    • 3 momentary pushbuttons - one for triggering the picture taking/scanning process, two for adjusting the audio volume level
    • 3 resistors (10 kOhm each) - not sure if these are needed here, but I keep them in to be on the safe side
    • 1 Adafruit MCP4725 Breakout Board - 12-Bit DAC w/I2C Interface (used as a replacement for the Pimoroni DAC board)
    • 1 Adafruit Mono 2.5W Class D Audio Amplifier - PAM8302 board (necessary to boost the audio signal to a usable level for the speaker)
    • 1 flat audio speaker
    • 1 NeoPixel stick (Adafruit NeoPixel Stick - 8 x 5050 RGB LED with Integrated Drivers)
    • 1 1N4001 rectifier diode (needed for the NeoPixel stick or strip)

    This setup should provide enough power for a mobile usage of the Raspberry Pi while also providing audio output for the Pi Zero.

    Wiring up the parts

    Here are the details for the part connections of the different parts mentioned above:

    Raspberry Pi GPIO:

    • 5V: connected to the PowerBoost 1000C 5V pin
    • GND(1): ground is connected to the PowerBoost 1000C GND pin
    • SDA: connected to SDA pin of the MCP4725 board
    • SCL: connected to SCL pin of the MCP4725 board
    • 3V: connected to VDD pin of the MCP4725 board
    • GND(2): connected to GND of the MCP4725 board
    • GPIO pin 18: connected to Din pin of the NeoPixel stick/strip
    • GND(3): connected to GND pin of the NeoPixel stick/strip
    • GPIO pin 23: connected to pushbutton 1 and resistor
    • GPIO pin 24: connected to pushbutton 2 and resistor
    • GPIO pin 25: connected to pushbutton 3 and resistor

    MCP4725 12-bit DAC board:

    • VDD: connected to Raspberry Pi 3V pin
    • GND: connected to Raspberry Pi GND
    • SCL: connected to Raspberry Pi SCL pin
    • SDA: connected to Raspberry Pi SDA pin
    • VOUT: connected to PAM8302 mono amplifier board A+ pin

    PAM8302A 2.5W mono amplifier board:

    • A+: connected to MCP4725 DAC board VOUT pin
    • A-: connected to Raspberry Pi GND
    • VIN: connected to Raspberry Pi 3V pin
    • GND: connected to Raspberry Pi GND

    NeoPixel Stick/Strip:

    • GND: connected to Raspberry Pi GND
    • DIN: connected to Raspberry Pi GPIO pin 18
    • +5V: connected to 1N4001 diode output side

    1N4001 Diode:

    • output: connected to NeoPixel +5V pin
    • input: connected to Raspberry Pi 5V pin (alternatively to Powerboost 1000C 5V pin)

    Powerboost 1000C charger board:

    • JST: connected to JST-connector of rechargeable battery
    • GND: connected to Raspberry Pi GND
    • 5V: connected to Raspberry Pi 5V and 1N4001 diode input
    • BAT: connected to left pin of slide switch (optional)
    • EN: connected to middle pin of slide switch (optional)
    • GND (2): connected to right pin of slide switch (optional)

    Optional/Temporary...

    Read more »

  • Slow but steady...

    Markus Dieterle03/21/2016 at 09:06 0 comments

    It's high time for another update on the progress for this project. The progress is slower than I'd like it to be, but at least It's moving in the right direction, or at least it seems to be for now.


    Hardware status

    With the exception of the Pimoroni pHat DAC audio board (which is only needed for the Pi Zero), I've now got all the additional components I need: switches, resistors, breadboards, a small speaker, the Powerboost breakout board, a rechargeable battery and - last not least - an additional RGBW neo pixel stick. I've added this last item in order to provide additional lighting for the camera when taking a picture. It's not as bright as a flash, but it should still be helpful for providing more contrast - that's another thing I'll have to test though.

    I also went ahead and soldered some standup pins to the 40 GPIO pins on the Rasperry Pi Zero, so I can connect it to the T-cobbler breadboard connector for testing:

    I might also get a small USB HUB for further testing which would allow me to connect the Pi Zero to my network while also connecting it to the USB webcam. For now, I can just switch over to the Pi 2 Model B for that.

    As for the Pimoroni pHat DAC audio board, that currently seems to be back in stock in the Pimoroni online store right now, but the supplies seem to be dwindling fast. I just checked my outstanding order in the online shop where I ordered the board a month ago, and that shows the board with an availability in 5 to 7 business days. So hopefully I'll get that not too long after the easter weekend.

    Just in case, I am also investigating other audio options for the Pi Zero.

    Ladyada already did a great job of recreating the basic audio hardware of the other Pi models for the Pi Zero and making it usable by using the GPIO PWM output with some hacking - you can read about the details here.

    Another option that in my mind should be viable here is the following:

    For the speaker audio output, I can use the same setup that Noel and Pedro Ruiz from Adafruit recently used for their Raspberry Pi PipBoy 3000 project. They connected the audio output of the Pi to the 2.5 watt mono amplifier breakout board and hooked that up to the flat mini speaker in order to have a direct sound output without using external speakers or headphones. This can directly be used for this project if we use a Pi Model A+, B+, 2 or 3.

    For the Raspberry Pi Zero, another component has to be added that provides the basic sound output. I'm currently thinking that a basic mini DAC breakout board like Adafruit's 12bit DAC board should work here. These can be hooked up to a microcontroller board or to a Raspberry Pi using I2C/I2S connection. There is already a short tutorial on the Adafruit Learning System on how to do this.

    By adding the "pulseaudio" software package to the mix it should be possible to "reroute" the standard audio output to the I2C/I2S-connected DAC board - at least in theory. I'll have to do some additional research and testing in this area.

    The only other option I can see right now would be to add a mini 2- or 3-port USB hub and an equally small USB sound card to the Pi Zero. If the components are small enough, and taken out of their default enclosings, they might not take up that much more space compared to the Pimoroni pHat DAC board, especially considering that they can be spaced out a little more freely.

    All things considered though, I guess the easy option would be to just use a different model of the Raspberry Pi instead (as all other Pi models already have a working audio hardware), and only add the mini amplifier and speaker. But then again, I usually choose the different, more complicated way in most areas because you just learn so much more - even if you fail.... and if it works, it's just great and should provide some benefits as well.


    Software status

    I put some thoughts into the software and created a simple framework for what my application needs to do, and then put that in a class. Rather than just write some single file code -...

    Read more »

  • Hard- and software progress, and a new idea

    Markus Dieterle03/07/2016 at 20:19 0 comments

    In the last few days I've been busy with other things, so the project did not get along as far as I would have liked.

    But I guess small steps will get me closer to the finish line as well as bigger steps, it just takes more time...

    So here's a little update:

    A little more hardware

    While a few pieces are still missing (or will likely be switched out like the pushbutton switches), I've managed to get hold of a few more hardware pieces for this project and started with the prototype assembly:

    The Adafruit Powerboost 1000 charger board has arrived, along with a rechargeable 4400 mAh lithium ion battery. The smaller 2200 mAh version wasn't in stock at the local reseller, as well as the smaller lithium polymer batteries. I don't have any data on how much power the webcam actually needs, but this battery should give me a pretty nice power reserve to work with.

    The PiMoroni pHat DAC board still hasn't arrived and it's unclear how long I have to wait for it, so for now I got the 12bit DAC board with i2C interface from Adafruit as an alternative. It should not be much more complicated to use this than the Pi hat version, and it's only needed for the Pi Zero anyway.

    Speaking of the Pi Zero: as you can see on the lower right side, the Pi Zero I won in the contest has just arrived today. I've also got a tiny USB to USB-2-go adapter, but no free USB hub at the moment. I can do some basic tests with the SD card I already used with the Pi A+ and Pi 2, using the adapter and my mini WiFi plug, but for complete tests I need to do some soldering.

    By the way, this is what the Pi Zero looks like with the power connection and the mini USB WiFi adapter:

    While the WiFi adapter almost vanishes inside a standard USB port, it still looks huge connected to the Pi Zero even with the tiny USB to USB-2-go adapter. This single board computer is really small... I like it! :)

    Slow software progress

    Due to my other activities in the last few days, the software development has suffered.

    I've started with coding, but did not make much progress so far. The good thing is, that the structure and details become clearer as I go along. Just like carving out a nice statue out of a block of stone.

    So nothing much to show here.

    I've also done some additional image processing tests and played around with different options and operations of the "graphicsmagick" software. Testing taking a full HD resolution greyscale image (1920 by 1080 pixels) from the C920 webcam in PNG format, I stumbled across another potential pitfall for the image processing: errors in the PNG to TIFF format conversion.

    Here is what one of the PNG files originally looked line (I had to export it to JPG as the PNG was bigger than the allowed 3MB):

    After just performing the PNG to TIFF conversion that is needed in order to get a valid input format for the tesseract OCR software, the result of the conversion looked like this:

    Needless to say, this does not work at all for a proper OCR conversion. I also tested the same image size and camera for a normal colour image, with the same result.

    Since the earlier tests did provide properly converted images, I guess this may have something to do with the additional "delay" and "skip frames" options I tried for "fswebcam". Strange nonetheless, as the default image looks pretty normal.

    Why can't the standard image processing be more reliable? :(

    A new idea - sort of

    Since the basic image quality is really important for the standard OCR processing, I've thought about different options for improving just that.

    Software optimisations aside, the best bet seems to be to use a different camera, with higher resolution, image stabilisation, autofocus etc. - all the nice features that most digital stills cameras already have, from simpler point-and-shoot cameras up to the big and expensive DSLRs.

    So it might be an even better idea in the long run to look into using a not-to-big point-and-shoot camera instead of a webcam.

    The "TextEye" could be a camera add-on, similar to an extended battery/grip...

    Read more »

  • More image tests and app preparation

    Markus Dieterle03/01/2016 at 17:53 0 comments

    Just a short update today...

    More image tests

    Using both the C270 and the C920, I took James Newton's idea and downloaded some street sign images. I also created another test document with some street names in really big font sizes and two slightly different fonts.

    After the usual image conversion using "graphicsmagick" the resulting TIFF images were processed by tesseract-ocr.

    Although I tried different image resolutions with the C920, and I also used the original downloaded images along with the photographed versions, most of the OCR conversions failed.

    I'll definitely have to do some additional research and testing here. Since I'm already using graphicsmagick for the image format conversion, I can also use this to perform some filtering and image enhancement.

    If I can find a nice GUI for graphicsmagick or imagemagick that runs with Debian Linux, I'll try to use that to find a good combination of image processing steps/options that will produce better OCR results. Once a good combination has been found, I can use this through the command line options of one of the tools. If there is no good GUI for both apps, I can switch to Gimp or something similar in order to find a good combination of image processing steps.

    I'll also take a closer look at image processing algorithms and see if there probably already is a good solution for this specific problem.

    Preparing the program

    Apart from the image and OCR tests, I am working on the structure of the main "TextEye" application. Once I'm satisfied with the basic flow, I'll start to convert that into actual code (hopefully starting tomorrow).

    Setting up this application and getting it to do what's needed seems easier right now than getting the image preprocessing right. I should be able to get some working result here even if the image processing is still off.

    More hardware

    I'm currently waiting on some parts I've ordered for this project. Sadly one of the parts - the Pimoroni "pHat DAC" board for the Raspberry Pi - isn't in stock right now, not even in Pimoroni's own web shop, and this stops the rest of the order being send out to me.

    I will try to at least get the rest of the parts I don't currently have by placing an additional order. Hopefully this will allow me to focus on the physical side of things later this week or maybe next week.

    Once the parts are halfway complete I can go about wiring the prototype, connecting the software to the hardware where necessary, and take some measures to develop a 3D design for an enclosure.

View all 16 project logs

  • 1

    Get a Raspberry Pi, SD card, webcam and other necessary hardware (see components list). You can ignore the additional components needed only for development.

  • 2

    Download the light version of Raspbian from the download page of the raspberry.org website.

  • 3

    Create a bootable SD card from the downloaded image.

View all 11 instructions

Enjoy this project?

Share

Discussions

granit xhaka wrote 10/06/2016 at 15:43 point

hi markus! I just found your project and I am working on a comperable project. this is the reason why i would really like to share some experiences with you! u mentioned that u made ur studies in karlsruhe: do u speak german? My goal is to build completly automatic glasses that can scan text and give it out as a audio. I am really sry for this confused text but I just wanted to contact u. and sry for my really bad english 

sincerely

granit

  Are you sure? yes | no

Markus Dieterle wrote 10/20/2016 at 14:48 point

Hello Granit,

I already send a reply, but maybe you did not recieve it somehow, so let's try again:

Sharing experiences is no problem...

...und natürlich können wir das auch auf Deutsch tun (wobei Hackaday dafür sicher nicht optimal ist).

Dein Projekt klingt in jedem Fall interessant und sollte sich im Wesentlichen mit den gleichen (oder vergleichbaren) Hard- und Software-Komponenten umsetzen lassen.

Wenn Du noch interesse hast, können wir uns gerne mal austauschen - ich bin zwar aktuell nicht allzu mobil, aber wir können uns ja auch über Skype, Google+ o.ä. austauschen oder einen Video-Chat starten.

Bis die Tage... :)

  Are you sure? yes | no

chris veigl wrote 03/25/2016 at 13:57 point

hi !  I tried to replicate your setup with fswebcam, tesseract and festival. I think the big problem is that the needed image quality for tesseract OCR can harldy be achieved in real-world conditions (especially when a person with a visual impairment takes the photo and cannot align the image / text orientation etc.) - Nevertheless I'm very much interested in your progress. 

Concerning the SD-interface to standard digital cameras: did you have a look at those WiFi-enabled SD cards (this could be a nice work-around without physically modifying the digital camera ...)

If it turns out that tesseract is still not happy with the image quality, maybe the online OCR solutions might be wort a look. I just experimented with a few - the best results gave the picture import into GoogleDocs and a nice service called "robobraille", see: http://robobraille.org/ - here you can convert to text or even directly to an mp3 file. 

  Are you sure? yes | no

Markus Dieterle wrote 03/31/2016 at 15:01 point

Hello Chris,

thanks for your interest.

You are right - all of the currently (freely) available OCR solutions were developed to be used with image input from flatbed scanners in mind. They work really good for that.

In my daytime job, I've also experienced the differences between normal flatbed scanners and special document scanners. The document scanners not only have a few additional functions for keeping differently thin papers in a stack separated during batch scanning, they also seem to be able to get more contrast and detail out of documents which have been degraded by being faxed back and forth a few times (so the text looks pretty blocky / fuzzy).

I guess most recent digital cameras should be able to produce better output than the webcams, and it's also possible to add polarizing filters.

Using WiFi-enabled SD cards could be a workaround, but you also would have to add a WiFi module to the Pi (unless you are using one of the new Pi model 3 boards). Even with just a small transmission distance, this increases the energy consumption for the camera and the Pi, cutting into the continous usage time. Might still be worth checking out.

Thinking about it - instead of modifying the camera for the SD card access, maybe the SD card itself can be hacked into. I haven't opened up a card yet, but I'm guessing that it should not be much more than a flash chip and some connection lines leading to the connector pins. If we connect some wires to the connection lines and carefully file away some of the plastic inside, it should be possible to connect the card either directly to the Pi or at least to an external, Pi-connected SD card reader. Then we can really just "hook up" the Pi to most cameras without having to modify them (although intercepting the "shot" signal would be nice). Ok, the doors for the SD card compartment (which usually also contains the battery of the camera) might be a litte difficult to close, but these might be replaced by custom 3D printed doors.

The "robobraille" service looks interesting, but this would also mean using WiFi or GSM internet connections, and possibly transferring some files with several megabytes each (unless you increase the JPEG compression rate, which might loose a lot of detail that would be useful for OCR processing). Also, you could not use this in places where wireless internet connections are flaky or not even present (which is still kind of normal outside of bigger cities).

Still it's a solid idea - and you could combine this with automatic translation for travels abroad (not perfect, but helpful)...

  Are you sure? yes | no

Robbie Murray wrote 03/21/2016 at 22:18 point

Hi. I actually own an insanely expensive device that does this.  It uses an unmodified Canon point and shoot.  I'm not sure of the model, but I could check.  It definitely grabs pictures directly via the USB connection. It is called the KNFB Reader if you want to look in to it.  It has now been replaced by an iPhone app.  It would be fantastic if this technology was more affordable.  Consider submitting this project for the 2016 Hackaday prise!  

  Are you sure? yes | no

Robbie Murray wrote 03/22/2016 at 10:16 point

I found a patent describing the only modification to the camera in the KNFB Reader - it has a polariser attached to the lens and the flash. The polarisers are perpendicular.  Apparently this improves image quality.

https://www.google.co.uk/patents/US7949245

  Are you sure? yes | no

Markus Dieterle wrote 03/31/2016 at 14:23 point

Hello Robbie,

Thanks for the input. I'll definitely take a closer look at that.

I will also try to take some pictures with my Panasonic TZ-5 camera and test those with the OCR. Since the camera can be connected to a computer via USB, this should also be possible for the Pi. 

The only downside is that the PC USB connection usually requires an additional mode selection on the camera - you have to choose between a normal PC connection (where the camera just connects like an SD card reader) or a PictBridge connection. I'm not sure if this can be surpassed.

But I think it should be possible to attach a simple SD card reader (breakout) to the Raspberry Pi and connect the wires from this to the SD card reader/writer inside the camera. Maybe a little complicated, but this should allow direct access to the SD card that the camera is using, without having to figure out how the reader is wired and controlled within the camera.

Anyway, since the OCR wasn't really developed for this kind of application, even good photos may require additional image preprocessing. We'll have to see...

  Are you sure? yes | no

Robbie Murray wrote 03/31/2016 at 21:50 point

The camera in the KNFB Reader is a canon PowerShot SD20.  It is connected via USB to a PDA ( I think it's an HP Ipaq, running custom software.  The correct USB mode is set with a switch.  It doesn't require an SD card, the images are sent directly over USB.

I couldn't find any way of taking pictures with it from linux.  Gphoto2 certainly has commands for taking a picture and downloading it, so it must work for some cameras.  Perhaps it would be worth asking the gphoto2 developers?

  Are you sure? yes | no

Markus Dieterle wrote 04/04/2016 at 14:25 point

I think the problem with the USB connection to "normal" digital photo cameras is that there is no live view image or video transmission over USB. Even on the most current cameras, you only get this as part of a WiFi or NFC connection, and there does not seem to be a standard in place, every camera vendor makes his own API functions for that (or so it seems). For USB connections, life view is not common, you usually get that only via HDMI (where you normally don't have a back channel for controlling the camera). It may work with an adapter that connects to a port that can take wire-based or wireless remote triggers.

With the webcams, we get a standard video stream that can be accessed in Linux as part of the device functionality. In order to have the same functionality for a "point-and-shoot" camera (or DSLM / DSLR), we would need a custom device driver and a special live view via USB function inside the camera. As the manufacturers don't release a lot of info, this could only work with a fully open source camera.

At the moment, the current open source camera projects - the Axiom camera (https://en.wikipedia.org/wiki/AXIOM_(camera)) and the Elphel cameras (https://en.wikipedia.org/wiki/Elphel) - seem to be optimized for different usage scenarios (professional film or security work), and the components are rather expensive, even more than a pretty good modern camera (like the LX100 from Panasonic, which has a terrific lens and sensor combination).

I've just checked out the "gphoto" project website - very interesting (even though not very modern in design :) ). The remote control cameras page is especially interesting (http://www.gphoto.org/doc/remote/). From what I've seen so far, some cameras seem to offer a direct USB or serial connection, while others only work in mass storage mode (where you only see the SD card when connecting it to your computer). The list (http://www.gphoto.org/proj/libgphoto2/support.php) is pretty impressive. I did not find the two models I currently own there, but they may yet be compatible (e.g. I have an older Panasonic Lumix DMC-TZ5 and there are some follow-up models on the list).

Before I go ahead and see if I can get a compatible camera used on eBay or elsewhere, I think I'll try to take some additional pictures with the cameras I have and transfer them to the Raspberry Pi so I can check if the OCR results can be improved this way.

  Are you sure? yes | no

James Newton wrote 02/26/2016 at 17:53 point

I'm not sure why the pictures didn't OCR well, but it might be the changes in intensity from one part of the page to the next which human eyes are so adept and removing. There is code for cleaning that up in the middle of my (somewhat sparse... just starting out) page on image recognition:

http://techref.massmind.org/techref/method/imgRecog.htm

http://www.bitquill.net/trac/wiki/Android/OCR (see the blog post) Given a greyscale image of dimisions m x n, the lighting will often change from one part of the image to another. To pick out details consistantly, you need to set the threshold individually for each pixel (i,j) as the mean intensity of pixels in a w x w neighborhood around the pixel. On each row, you might do 
for i = 0 to N-1: s = 0; for k = max(i-r,0) to min(i+r,N-1): s += a[k];
where w=2r+1. (r is half the width of the window). But it turns out that this is faster:
Initialize s = sum(a[0]..a[r]); for i = 1 to N-1: if i > r: s -= a[i-r-1]; if i < N-r: s += a[i+r] 
at the end of each, s is the threshold for pixel a[i].

  Are you sure? yes | no

James Newton wrote 02/26/2016 at 16:11 point

May I suggest some testing with text laid out in a street sign size and format? My understanding is that there are existing scan / ocr / read setups for things like books and documents, but I've not see something that is portable and as cheap as this would be. The ability to read signs "in the wild" would be a gift, and the lower resolution picture, with larger letters might work better.

  Are you sure? yes | no

Markus Dieterle wrote 02/26/2016 at 16:50 point

Hello James,

Thanks for your input.

That's definitely something I can try. I guess without a camera that can automatically focus (and maybe even zoom) in a way that produces good images without manual intervention from the user, this is a usable option.

It would be nice to have something more versatile that could also help with reading things like a menu in a restaurant, but this does not seem to work properly so far with a USB webcam.

Even with street signs with sufficiently big lettering, a lot of testing remains to be done in order to determine the best distance for taking images and maybe reduce the influence of slight shaking.

It would be great to use a good and sufficiently cheap digital photo camera instead of the webcam, but I don't know about one that would offer live image view via a USB connection - or an option to "hack" that in somehow.

Anyway, I'll try to come up with some proper street signs for further testing.

Thanks again for your interest in this project.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates