Close
0%
0%

Unfolding Space

see with your hands

Similar projects worth following
The “Unfolding Space” project deals with sensory substitution: A phenomenon by which the function of one missing or dysfunctional sensory modality is replaced (substituted) by stimulating another one. The outcome is a prototype that projects a 3D picture, generated by a special camera, as vibration patterns on the back of the hand.

The blind can therefore use their tactile modality of the hand to actually see.

TECHNICAL DETAILS AND CODE will be published soon!

This Project started back in 2018 as my BA Thesis including a more theoretic and scientific research paper about sensory plasticity (find them under "links" but in German only, sorry). After finishing that first step, I constantly developed the device further and as the project was published open source, I decided to show my progress here. Feedback is very welcome...

Basic Functionality

A 3D generated picture from a depth camera is haptically projected on the back of the hand by using vibration patterns. The location of a vibration depicts an object’s relative position in space, the strength of the vibration represents its distance.

Scientific Background

The theoretical background of this project is called Sensory Substitution. A phenomenon by which the function of one missing or faulty sensory modality is replaced (substituted) by stimulating another one – in this case the tactile modality. At the beginning of the substitution process, this new stimulation has to be actively interpreted by analysing the tactile stimulus. But after some training the new visual-like Input becomes implicit and gets processed subconsciously. Users begin to see the space in front of them.

Motivation

These scientific efforts started more than 50 years ago, but even today there are almost no blind people using substitution processes to handle the absence of their visual system. All attempts to develop a device for the broad market failed.

Process

Many projects failed on a practical implementation. An analysis revealed that despite the elaborate technology used by them, design and usability issues were often not taken into place. Therefore I followed an open and iterative rapid-prototyping approach to quickly work out strengths and potentials and to identify limitations of the hardware and algorithms. Even though my theoretical work about Sensory Plasiticity had already predicted many aspects, it was this process of prototyping that lead me to a functioning device that quick.

  • 1 × Raspberry Pi 3b+
  • 1 × PMDtechnologies Pico Flexx Time-of-Flight Camera
  • 9 × DRV2605l Drivers for LRA Actuators
  • 1 × TCA9548A I2C Multiplexer
  • 1 × G0832012 LRA Vibration Actuator

View all 9 components

  • Raspi Slimdown, New Case

    Jakob Kilian06/16/2019 at 16:04 0 comments

    Short update on my progress:

    To make everything smaller and more mobile, I designed a new case with a ultra slim 10000mah powerbank from TNTOR. As I didn't need the USB and Ethernet sockets I slimed down my Raspi as you can see on the pictures. The new casing is now already quite neat and slim but the elastic band with velcro fastening didn't work out as expected. It is a little bit too limp and has too less adjustability to different arm. I might have to come up with a different solution here. 

  • Intermediate: PCB for driving the 9 vibration actuators

    Jakob Kilian06/08/2019 at 11:38 0 comments

    The bulky self made board out of 9 DRV2605l breakouts can no go into retirement. I successfully planned, ordered and equipped PCB is finally there and – hoooray – it works.

    It took me some efforts to learn how to design a proper circuits and even more to reflow the tiny parts (thanks to my friends at Presseverykey). But it seems like I succeeded. Most of you may be familiar with circuit boards – to me at least it was a great excitement to see it working in the end ;-)

    Next Step: As I couldn't find a worthy slim/small alternative single chip computer I decided to dismantle all the ports of the Rpi3b+ to finally fit everything except the battery directly IN the glove. Lets see how this works...

  • Debug App (via UDP - Wifi) and Fraunhofer Institute

    Jakob Kilian05/15/2019 at 10:18 0 comments

    For a testing Session at the Fraunhofer Society I developed my code further and implemented the possibility to check all the values of the raspi/glove in real time. 

    After some struggles with a UDP library for C++ (boost –> asio) I managed to get a relatively stable connection to my Android App (written in Processing) over a Wifi hotspot opened by the very same phone. This was an important step to me, as I now could monitor the status and the values of the app, see possible crashes of the royale library and also see the visual "perspective" the glove-user actually "sees". The app is in German right now, but will be english in future...

    With this stuff I went to Berlin to the Open Communication Systems – FOKUS research Institute of the Fraunhofer Society (budgetwise the biggest organization for applied research in Europe). There I met Jonas Willaredt (project manager of an indoor-navigation research project) and had the chance to firstly talk about my concept and its progress and secondly to test my prototype with an experienced tester of inclusive media – Stephan Heinke, who is blind himself. Eventhough it turned out (once again) that the device is not jet on level of development to be a useful navigation aid for blind people, the test and the conversation still led to a lot of productive feedback for me. Hopefully this won't be our last meeting, as the institute can in principle imagine to support the project in an adequate form. 

  • Stability Issues, Raspi 0 Disappointment

    Jakob Kilian05/06/2019 at 08:39 2 comments

    This is a short update of my project.
    End of this week I will have a short meeting / testing with a big german research organization – therefore I am working again on stability issues.
    The "royale" library from PMDtec crashes after some time so I build a mechanism that checks the activity of the library and reinitializes it after inactivity. Restart now also happens, when camera gets detached accidentally – the whole starting procedure furthermore just takes about 5 seconds now...
    Currently I am building a interface for android phones to check the status of the program. It is getting better and hopefully I am able to finally upload the code after this session. 

    Furthermore I tested the Raspi Zero W with my Code. Didn't know that it is just a ARMv6 processor, so it didn't work due to NEON commands off the "royale" library. Still looking for an powerful alternative smaller to RPi 3b+ to save energy and size. Thinking about ODROID C0 but it doesn't seem to be a widely used board... 

  • Intermediate: new case, new hardware assembly, new code

    Jakob Kilian02/16/2019 at 10:13 0 comments

    Having an upcoming TV shooting with a blind test person in mind, I reached another intermediate goal: as already mentioned in my previous post I moved the platform from laptop to Raspi 3b+ and from crappy plugged electronics to a soldered shield doing all the output stuff. Things can be much smaller in future being soldered on a pcb made for this purpose. But for now I used the breakout boards of the TCAs and the DRVs as it is easier to solder them and to replace them in case of faults.

    Power Supply

    I decided to use standard USB powerbank for power supply as they are interchangeable, they already come with housing, charge controller and some safety features. As my nine vibration actuators take around 400mA at 3V I decided to power them from the raspi’s 3V output pin. Therefore I can power the whole device with a  powerbank. As one read in many discussions on the internet, voltage is a critical thing for the 3b+. Using bad USB cables (and most of them out there are indeed bad) voltage can drop significantly under full load. That is because powerbanks are not made to keep the voltage on a specific level and those cables come with very thin wires (some of mine had more than 0.2 ohm on 20cm which would lead to -0.4v at 2A). As it is difficult to find cables that have specific data about their quality (like the AWG value) I just ordered a few to test. In the end I chose a 10cm cable from System-S and got a the Anker PowerCore Lite 10000mAh which comes with 5.2V output (little bit higher than others) and which is said not to have huge voltage drops. With this setting I have relatively stable voltage supply and didn’t get the voltage-warning-bolt.

    Range Adjustment

    As the use cases of the device differ a lot (e.g. indoor vs. outdoor) I wanted to have a adjustable depth range. As I needed an interface which is useable for blind people, I went for an analog potentiometer. It is input and output at the same time as it can be used both to adjust

    the value as well as to read the current range setting. Using an analog poti with a raspberry is more difficult than I thought, as you need a separate a/d converter because of the absence of analog inputs on the raspi. The Digispark microcontrollers are a good workaround for this (thanks to: Paul C. Brown) I had some trouble setting it up but in the end it worked out pretty well. For a better tangibleness I printed out an asymmetrical poti cap.

    Shield and Case

    The soldering of the shield and the mounting on the raspi wasn’t planned very well but included a lot of improvising. As I wrote in the beginning, this is just an intermediate step and the ICs can be placed in much higher density on a single pcb in future. When everything was fixed I lasercut an acrylic case to protect the electornics from shorts and pressure whilst ensuring enough air flow to protect the raspi from overheating. Everything was crammed in the cheap sport arm band you can see on the pictures to wear it at the upper arm. The camera on the glove also got a new 3d printed support:

    Code

    I am designer and therefore not an expert in coding. To port the code to the raspberry I I therefore got help from friends at Press Everykey. After loads of performance and stability issues I added the adjustable range function and the possibility to stream the raw tof camera data to a tablet as well as the processed tiles that control the vibration actuators. As I already announced, I will release the code, when it is on a clean and understandable level. If you are interested in hundreds of messy code lines you can of course contact me now… ;-)

    Shooting 

    After replacing a broken actuator in the last night, the stuff was finally ready for the shooting with the german tv station. Technically everything went well but due to a fix schedule there wasn’t much time to go into detail. The blind test person had problems to get familiar with the new device as we didn’t really have the possibility to train the new inputs...

    Read more »

  • Moving the Platform...

    Jakob Kilian02/08/2019 at 11:03 0 comments

    The first prototypes of this project started with a laptop in the backpack. This had many disadvantages like overheating, weight and a lack of comfort. Also the code was very buggy in the beginning, running into crashes that I couldn't reconstruct.

    So I started moving the project to a new platform (Raspi) and overworking the hardware and code. I am now at a much more portable and stable point and I will release more infos and hopefully the code (when I cleaned up my beginner-level writing) very soon.

    For now here are some progress photos:

View all 6 project logs

Enjoy this project?

Share

Discussions

Jakob Kilian wrote 02/15/2019 at 12:53 point

thanks guys. Just wanted to add: the basic idea behind that is kind of old – check out Paul Bach-y-Ritas amazing studies back in the 1960s (https://youtu.be/7s1VAVcM8s8?t=147) and loads of others (keyword sensory substitution). New is the 3D tof technology and cheap hardware in general. This brings everything on a new level...

  Are you sure? yes | no

Josh Cole wrote 02/15/2019 at 06:38 point

This is such a cool idea! I've been thinking recently about how to use other sensory mechanisms beyond eye sight in order to convey information. This is next level though, and could be life changing for some people. I can't wait to see/feel where it goes next!

  Are you sure? yes | no

Malek wrote 02/12/2019 at 19:24 point

great work keep going

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates