close-circle
Close
0%
0%

Visioneer

AI glasses that provide traffic information and obstacle avoidance for the visually impaired.

Similar projects worth following
close
Worn as a pair of sunglasses, the ‘Visioneer’ will provide timely traffic information and obstacle detection to the visually impaired. A trained neural net will provide a level of speed and accuracy necessary for real-time recognition and response. Our design will strive to maximize ease-of-use, comfort and reliability to supplement the user’s existing navigation options and “feel for the world”.

GitHub:  https://github.com/MakerVisioneer/prototype

Google Drive: https://drive.google.com/drive/folders/0B3wmJbFj6cTCTGMzemFlTmdGbnc?usp=sharing

The problem

Those of us who have never struggled with a vision impairment beyond a need for corrective lens may take for granted the ability to see obstacles, gauge oncoming vehicles before crossing a street or find the entrance to our favorite coffee shop.  These everyday activities can be challenging and intimidating for the visually impaired.  Guide dogs are one option to provide this additional guidance. However, access, cost, maintenance and/or allergies may make ownership impractical.  'Guiding Eyes for the Blind' estimates that “only about 2 percent of all people who are blind and visually impaired work with guide dogs.” (“Mobility” https://nfb.org/blindness-statistics)  

Assistive canes have their own limitations.  While useful to detect low-level obstacles and walls, a cane cannot detect head-to-chest level obstacles (i.e., tree branches).  Assistive canes cannot determine the location of entryways without direct contact, identify objects, or detect traffic condition.

There are wearable devices in or near market to address some of these issues, but their costs are in the range of $2000.

How Visioneer works

The Visioneer, appearing as a set of sunglasses, will perform vehicle detection using two cameras and a combination of OpenCV and a local neural net to recognize objects in the user's path.  It will provide user feedback via a bone conductor, without interfering with their ability to hear normally.  

Usage Flow Diagram 

To illustrate how Visioneer works, we drew the flowchart shown below. The key here is to first determine whether the user is walking or stationary.  This makes a difference in how the user interacts with their surroundings and decision making.  When the user is walking, Visioneer's obstacle avoidance ability will come into play.  When the user is stationary, that signifies to Visioneer that the user either is trying to identify something at a near distance or waiting to cross the street.  The easiest way to determine the user's situation would be to use speech recognition but considering its unreliability and potential social awkwardness, we decided to go with other options that include the combined use of software and hardware components.

Schematic (First Draft)

Based on the usage flow diagram, we decided to use an accelerometer to determine if the user is walking or stationary.  We use OpenCV to perform obstacle avoidance. To determine if the user wants to identify something at a close distance, we use lidar.  If the user is stationary and isn’t close to any objects, OpenCV and a local neural net will identify surroundings to determine if the user is looking at traffic or other objects. Everything will operate on a Raspberry Pi Zero. 

Visioneer3DPrintSet.stl

3D modeling file of Visioneer's modular parts.

Standard Tesselated Geometry - 644.42 kB - 09/04/2017 at 06:32

download-circle
Download

Visioneer Assembly Diagram.pdf

Visioneer assembly parts list and diagram

Adobe Portable Document Format - 114.11 kB - 09/01/2017 at 02:04

eye
Preview
download-circle
Download

Traffic detector.jpg

Learning software techniques to implement traffic detector. Here are the three techniques.

JPEG Image - 267.01 kB - 09/01/2017 at 01:48

eye
Preview
download-circle
Download

3d design.png

initial 3d design.

Portable Network Graphics (PNG) - 84.93 kB - 08/18/2017 at 14:53

eye
Preview
download-circle
Download

3d design labels.jpg

Initial 3d design with components labeled and described

JPEG Image - 437.69 kB - 08/18/2017 at 14:52

eye
Preview
download-circle
Download

  • 2 × Raspberry Pi Zero W
  • 1 × MPU 6050 (not in phase one build) 9DOF Sensor
  • 1 × TCA9548A Multiplexer Switches and Multiplexers / Analog Switches and Multiplexers
  • 2 × Bone Conductor
  • 1 × VL53L0X (LIDAR) Lidar

View all 14 components

  • Experiment on OpenCV, sensors, and bone conduction transducer

    MakerVisioneera day ago 0 comments

    OpenCV experiment on color and circle detection with a traffic light picture. These techniques will be used to detect traffic lights for the traffic detection in Visioneer. Other OpenCV techniques will be further used as well for traffic detection.


    Sensor experiment

    Here are two videos of  the experiment on LIDAR and MPU6050. The first video is testing two TOF LIDARs VL53L0. The conclusion is stable readings and narrow detection range. So we've decided to experiment with Maxbotix Sonar sensor for obstacle avoidance.

    The second video is using MPU 6050 to detect movement of a user. Turning on red LED light means a user is stationary and turning on blue LED light means the user is walking or moving. It is a way to switch between two modes in Visioneer.

    Testing audio output on bone conduction transducer

  • Deep Learning Experiment 1

    andrew.craton3 days ago 0 comments

    The first video shows my first experiment of live recognition using a Pi3, USB webcam, Movidius\Caffe and OpenCV on a pre-trained neural net called SqueezeNet.

    The second video shows the same Pi3  setup classifying a single cat pic at 307ms.  Hopefully you can see in the video the number 307523 which is = 307ms.

    Now that we have a successful benchmark for Movidius, we will turn our efforts to benchmarking without it, and using YOLO\Darknet instead of Caffe\Squeezenet.

    After that, we will train\compile a custom neural net on traffic-related images which should achieve a faster recognition speed on either platforms.  The goal is < 40 ms on a small custom set of objects.

  • (5) Research: Accessible Pedestrian Signals

    MakerVisioneer09/04/2017 at 04:53 0 comments

    The National Cooperative Highway Research Program Project 3-62, Guidelines for Accessible Pedestrian Signals highlights the procedure used by the visually impaired to cross an intersection.  

    http://www.apsguide.org/appendix_d_understanding.cfm

    We've outlined the procedure and how Visioneer could help.  Visioneer's phase two design will be implemented based on the outputs of the flowchart below.

  • (4) Testing Visioneer's phase I design with software implementation

    MakerVisioneer09/04/2017 at 00:19 0 comments

    Testing Visioneer's phase I design implemented with OpenCV in cross traffic.

    https://www.youtube.com/watch?v=74L4lV5V-yQ&feature=youtu.be

    Visioneer's phase II design will focus on improving traffic detection and accessibility with neural nets , machine learning, deep learning, or other open source AI frameworks.

    The Pi Zero doesn't have the hardware to support sophisticated neural nets, so we are designing custom datasets which will be minimal in size and complexity.  We are also optimizing OpenCV to maximize the camera FPS.  TinyYOLO\Darknet is our current open-source choice of local neural net, but we also have a Movidius (Myriad 2) USB ($79), which can process Caffee 1 networks at around 100 gflops, while only using 1W of power.

  • (3) Assembly of 3D printed parts

    MakerVisioneer08/30/2017 at 00:22 0 comments

    Unfortunately we found the glasses and hand band design was not comfortable to wear.  So we decided to go with a design that incorporates 3D printed modular parts that snap onto existing sunglasses.  This design is much more comfortable and we think it will be more appealing to potential users. 

    Here is an assembly diagram with the 3D printed parts.

  • (2) Initial 3D design and 3D prints

    Debbie Leung08/18/2017 at 23:04 0 comments

    We have considered several different design options for Visioneer such as a bracelet, necklace/pendent, pin, glasses, headband, and assistive case attachment.  Each option has both pros and cons.  Finally we narrowed it down to a design involving both glasses and a headband.  The glasses allows the user to "look around" for traffic and provides a good height for obstacle avoidance.  The headband will hold the power source and help to balance out the overall weight.  This design also allows us to use a bone conduction transducer that provides the best audio signal reception and vibration feedback. 

    Based on the component list, schematic, and results of camera experiment, we've come up with the following initial 3D design.


    We 3D printed the glasses frame and headband to test for comfort and wearability.

  • (1) Exploration and Experiment

    MakerVisioneer08/16/2017 at 18:20 0 comments

    We’ve explored the recognition capabilities of Google Vision and OpenCV. We decided to go with OpenCV, for our phase one design, because of its movement detection capability.  Google Vision remains an option for recognition, but we intend to try local neural nets first to eliminate the dependency on an outside service.

    Recognizing the label with Google Vision when it is less than 20 cm.

    Using OpenCV to detect motion, speed, and position with a selected area on a frame.

    Google cloud vision's activation by detecting an object with more detailed info within 30cm by an Ultrasonic Sensor

    Google Vision and Speech API via Asus TinkerBoard

    Motion tracking with OpenCV

    We also did an experiment to determine how many cameras should be used, as well as, which angle is best for traffic detection. The conclusion was that two cameras in the angle of 60 degrees are best for performing traffic detection. 

    1. One camera, forward facing, no deflection.

    2. Two cameras, forward facing, no deflection

    3.  Two cameras, 15 degree deflection

    4.  Two cameras, 30 degree deflection

    5.  Two cameras, 45 degree deflection

    6. Two cameras, 60 degree deflection

    7. Two cameras, 90 degree deflection (side facing)

View all 7 project logs

  • 1
    Assembly of Visioneer frame to pre-wired components

    Video walks user through assembling a set of components onto one side of the prototype frame.  NOTE: we have transitioned away from using the sonar, so you will see only the LIDAR in this video. Also, the power carrier position has changed to behind the main carrier (see current gallery), rather than next to it, as shown in this video.

  • 2
    Quick OpenCV installation

    Claude Pageau's instructions: https://github.com/pageauc/opencv3-setup

View all instructions

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates