Close
0%
0%

Onero - Sign Language Translation Device

The focus of this project is to build a low cost, minimalist sensor glove that can be used to translate Sign Language for the Deaf community

Similar projects worth following
According to WHO, over 5% of the world's population suffers from disabling hearing loss. That is over 360 million people whose lives are affected and these people suffer from a communication breakdown. This breakdown can have a large impact on the persons life, from education to social and emotion development.

The Onero glove is a low cost device that can be used to translate sign language and help to bridge the communication gap between the hearing and Deaf communities.

The Hackaday Prize tag line is "Build something that matters" and this project matters to me greatly and I believe that it could be a great help to many people!

The project will involve developing a minimilistic glove that uses IMU's to measure finger and hand movements, bluetooth for wireless communication and feedback for correct gestures.

Hand tracking and gesture recognition also has applications in VR, health and home automation!


Concept design Details

The Challenge:

According to the World Health Organisation, over 5% of the world’s population suffers from disabling hearing loss. That means that over 360 million people are at risk of not being able to communicate effectively with those around them. In turn, the Deaf often struggle to gain access to employment and to interact socially with their peers. Research conducted in Bulowayo, Zimbabwe suggests that children with severe hearing impairments not only have low reading levels and limited vocabulary, but also that they often feel a sense of isolation and loneliness (Mpofu and Chimhenga 2013). These challenges are also well documented here: http://www.who.int/mediacentre/factsheets/fs300/en/.

Through the application of innovative technologies, I believe that the challenge of effective communication between hearing and Deaf communities can be overcome. That is why I am taking on the challenge of developing and producing a device that can record and translate the hand motions used in South African Sign Language. Because I am from South Africa, I will develop the device with local Deaf communities in mind. It will however be made to be easily transferred to different contexts.By translating Sign Language in real time, the device will allow Deaf people to easily communicate with those who do not know sign language and vice versa. The technology can also be used by non-signers to help them learn Sign Language. It will be especially valuable for parents and educators of Deaf children.

In order to add value to the everyday lives of both Deaf people and those with whom they interact, this technology will have to be accurate and integrate seamlessly into their daily routines. The intention is also for the device to be worn all day. Given these aims, the device must meet the following criteria:

  • Light weight - The device must be comfortable and easy to wear. If it is heavy, the device will add to the “gorilla arm syndrome” that has been associated with data glove and gesture recognition devices.
  • Minimal and sleek design - The device should look good and be ergonomic.
  • Low power/Long battery life - To be useful the device should at the very least have a 12 hour battery life so that the user does not have to charge it throughout the day. They should be able to put it on when they wake up and only have to charge it again when they go to sleep that night. Ideally the device should last several days on a single charge.
  • Must not impede daily activities - A user should barely notice that the device is there. This will be a very important feature, since ease and comfort of use will impact on the user’s willingness to communicate through the device. The goal will be to design a device that mimics the sensation of wearing some rings and a wrist watch.
  • Reasonable cost - In many parts of the world, particularly developing countries like South Africa - poverty intensifies the negative impacts of hearing impairment. Because my intention is to build something that will bring about significant change in the lives of those who struggle to access adequate education and employment, the device will have to be accessible to as many people as possible. This means focusing very hard on cutting out any unnecessary features. The device must be kept as lean as possible.
  • Wireless communications - The device needs to be able to connect with a more powerful processing unit in order to do the translations. Having the device tethered with a wire would be awkward and severely limiting. By using bluetooth (BLE) the device will be able to connect to a smartphone with ease.
  • Quick and Accurate translation - Above all else this is the most important criteria. If the device does not translate quickly enough, it will stunt communications and cause frustration. If it does not translate accurately it will cause confusion and would defeat the purpose of the whole device.

My Background

At this point in time the team consists of myself, Ben McInnes. I...

Read more »

  • 1 × Teensy LC
  • 6 × LSM6DS33 The LSM6DS33 is a system-in-package featuring a 3D digital accelerometer and a 3D digital gyroscope performing at 1.25 mA (up to 1.6 kHz ODR) in high-performance mode and enabling always-on low-power features for an optimal motion experience for the consumer.
  • 1 × Bluefruit EZ-Link Bluetooth Module

  • Sensor Ring Render

    mcinnes.bs04/25/2016 at 05:07 0 comments

    Here is a quick render of what the ring might look like.

  • Simple Concept Sketches

    mcinnes.bs04/24/2016 at 21:46 0 comments

    I am not very good at sketching but here are two pictures that give an indication of my idea.

    The first image is just to show the layout of the device. Each finger will have one IMU ring that connects back to the main unit via cables. The IMU rings will be placed on the middle phalanx of each finger.

    The Second image shows a simple concept picture of the IMU ring. The body will be 3D printed and it will house the IMU on the top. Another feature to consider is adding padding to the inside of the ring to make it more comfortable and to make sure it is secured to the finger really well.

  • Initial System Design

    mcinnes.bs04/24/2016 at 21:17 0 comments

    Hardware

    The hardware used in this project will grow and evolve as the design improves. The hardware will be developed in three main stages: Proof of concept, prototype and finally (if time and budget allows for it) custom boards.

    Proof of concept

    Unfortunately due to budget constraints and the weak South African Rand, I am limited to creating a single glove at this point in time. The proof of concept will be made using off the shelf components and will most likely be held together with duct-tape. I am hoping to have access to a 3D printer to create housings for the components.

    The basic components that I will use for the proof of concept are as follows:

    Microcontroller: Teensy LC (https://www.pjrc.com/teensy/teensyLC.html)

    The Teensy will do some basic processing, handle the communications (both bluetooth and the collecting data from the sensors) and do power management. The Teensy is great because it is powerful, low cost and can be programmed using the Arduino IDE which will help to speed up prototyping time.

    Inertial Measurement Unit: 6DoF IMU board (LSM6DS33)

    Six IMU’s will be used in this device; one for each finger and one at the back of the hand. Ideally 9DoF devices would be used to help reduce drift in the gyro readings but at this time it is out of the budget. THe LSM6DS33 was chosen for its very low power consumption and “Always On” feature which will be very useful in this application.

    Communication: Bluefruit EZ-Link Bluetooth Module

    Bluetooth works great because it is low power and can communicate easily with both PC’s and smartphones. This is important as most of the processing is going to be done off the device in order to save power and reduce cost.

    The total came to R2305 which is about $162. This is the bare minimum for one of the gloves.. As mentioned this is for the proof of concept and so the device will be tethered to a desktop power supply. Once the design has proven successful, the necessary components will be added to cut the tether. A fully fledged prototype will be tested.

    Prototype

    The prototype will be an extension of the proof of concept, it will need to meet the criteria laid out in the previous section. It is likely that there will be several iterations of the prototype by the end of this project. As those iterations are created I will be sure to update this section.

    Software

    The device will be programmed using the Arduino IDE and try to use available open-source libraries instead of trying to reinvent the wheel. The goal is to get up and running as soon as possible. My goal is to try to make the code modular so that it will be easy to maintain and change as the project evolves.

    The Repo will be made available as soon as the basics are up and running. Any comments or advice is always welcome! I am always happy to hear constructive criticism because I am always eager to learn something new.

    As I see it the software will be broken up into the main functions. The firmware will have several main functions: calibrating the device, capturing the user data, transfering the data to another device.

    There will need to be software that can receive and store data from the Onero device, once this is done the software needs to use machine learning in order to translate the data into words. A GUI would be useful to help visualise the data as well as to display the data as a digital hand.

    Gesture Recognition Approach

    Raw Data: The data needs to be collects in a precise and organised manner. It will require several fluent signers as well as a few learner signers to ensure that the models can generalise well. Each signer will have to do several sets of tests. These tests will start with simple static gestures (like the alphabet) and then move on to dynamic gestures. The dynamic gestures will also start with single words and then move on to full sentences and conversations. The collection of data will be the most time consuming part of this project and will need to be very clearly thought out in order to make sure that...

    Read more »

  • Components have arrived!

    mcinnes.bs04/24/2016 at 20:16 0 comments

    So after a few delays and confusion with the order, I have finally gotten my hands on the components I need to build the proof of concept!

    Much excite!

View all 4 project logs

Enjoy this project?

Share

Discussions

Bhavesh Kakwani wrote 09/19/2016 at 17:56 point

Great idea. I wanna mention that using an IMU to get position information creates a lot of drift error, so make sure to find a way to mitigate that!

  Are you sure? yes | no

silvio biasiol wrote 03/28/2016 at 00:02 point

good luck! That's a nice project!

  Are you sure? yes | no

masa wrote 03/15/2016 at 17:59 point

Amazing, hope this gets made!

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates