Close
0%
0%

Unfolding Space

see with your hands

Similar projects worth following
The “Unfolding Space” project deals with sensory substitution: A phenomenon by which the function of one missing or dysfunctional sensory modality is replaced (substituted) by stimulating another one. The outcome is a prototype that projects a 3D picture, generated by a special camera, as vibration patterns on the back of the hand.

The blind can therefore use their tactile modality of the hand to actually see.

TECHNICAL DETAILS AND CODE will be published soon!

This Project started back in 2018 as my BA Thesis including a more theoretic and scientific research paper about sensory plasticity (find them under "links" but in German only, sorry). After finishing that first step, I constantly developed the device further and as the project was published open source, I decided to show my progress here. Feedback is very welcome...

Basic Functionality

A 3D generated picture from a depth camera is haptically projected on the back of the hand by using vibration patterns. The location of a vibration depicts an object’s relative position in space, the strength of the vibration represents its distance.

Scientific Background

The theoretical background of this project is called Sensory Substitution. A phenomenon by which the function of one missing or faulty sensory modality is replaced (substituted) by stimulating another one – in this case the tactile modality. At the beginning of the substitution process, this new stimulation has to be actively interpreted by analysing the tactile stimulus. But after some training the new visual-like Input becomes implicit and gets processed subconsciously. Users begin to see the space in front of them.

Motivation

These scientific efforts started more than 50 years ago, but even today there are almost no blind people using substitution processes to handle the absence of their visual system. All attempts to develop a device for the broad market failed.

Process

Many projects failed on a practical implementation. An analysis revealed that despite the elaborate technology used by them, design and usability issues were often not taken into place. Therefore I followed an open and iterative rapid-prototyping approach to quickly work out strengths and potentials and to identify limitations of the hardware and algorithms. Even though my theoretical work about Sensory Plasiticity had already predicted many aspects, it was this process of prototyping that lead me to a functioning device that quick.

  • 1 × Raspberry Pi 3b+
  • 1 × PMDtechnologies Pico Flexx Time-of-Flight Camera
  • 9 × DRV2605l Drivers for LRA Actuators
  • 1 × TCA9548A I2C Multiplexer
  • 1 × G0832012 LRA Vibration Actuator

View all 9 components

  • I Reached Hackaday Price Finals - Yeah!

    Jakob Kilian5 days ago 0 comments

    Hey together,

    this will be a short log, as I am a bit busy and as I'm working hard on the preparation for the Hackaday Price Finals 2019 which I have been nominated for. In the final round there are 20 amazing projects – all of them have huuuge potential and some of them are indeed very advanced and technically complex. Therefore I feel very honoured to be nominated with a more interaction-, design- and concept-focused project. 

    Furthermore I won a Mark in the Taiwanese Golden Pin Concept Design Award and I run for the "best ofs". In this connection I got also featured on designboom.com which I am extremely happy too. :-)

    Last but not least I wrote an application for a research assignment together with my university. If this will work, I could dive deeper into the research and hopefully speed up the process..

    These are the reasons why I am currently producing video footage for the documentation like this interview here ;-)

    There is still a lot to do, but there will be news very soon. Thanks for your support!

    Jakob

  • Raspi Slimdown, New Case

    Jakob Kilian06/16/2019 at 16:04 0 comments

    Short update on my progress:

    To make everything smaller and more mobile, I designed a new case with a ultra slim 10000mah powerbank from TNTOR. As I didn't need the USB and Ethernet sockets I slimed down my Raspi as you can see on the pictures. The new casing is now already quite neat and slim but the elastic band with velcro fastening didn't work out as expected. It is a little bit too limp and has too less adjustability to different arm. I might have to come up with a different solution here. 

  • Intermediate: PCB for driving the 9 vibration actuators

    Jakob Kilian06/08/2019 at 11:38 0 comments

    The bulky self made board out of 9 DRV2605l breakouts can no go into retirement. I successfully planned, ordered and equipped PCB is finally there and – hoooray – it works.

    It took me some efforts to learn how to design a proper circuits and even more to reflow the tiny parts (thanks to my friends at Presseverykey). But it seems like I succeeded. Most of you may be familiar with circuit boards – to me at least it was a great excitement to see it working in the end ;-)

    Next Step: As I couldn't find a worthy slim/small alternative single chip computer I decided to dismantle all the ports of the Rpi3b+ to finally fit everything except the battery directly IN the glove. Lets see how this works...

  • Debug App (via UDP - Wifi) and Fraunhofer Institute

    Jakob Kilian05/15/2019 at 10:18 0 comments

    For a testing Session at the Fraunhofer Society I developed my code further and implemented the possibility to check all the values of the raspi/glove in real time. 

    After some struggles with a UDP library for C++ (boost –> asio) I managed to get a relatively stable connection to my Android App (written in Processing) over a Wifi hotspot opened by the very same phone. This was an important step to me, as I now could monitor the status and the values of the app, see possible crashes of the royale library and also see the visual "perspective" the glove-user actually "sees". The app is in German right now, but will be english in future...

    With this stuff I went to Berlin to the Open Communication Systems – FOKUS research Institute of the Fraunhofer Society (budgetwise the biggest organization for applied research in Europe). There I met Jonas Willaredt (project manager of an indoor-navigation research project) and had the chance to firstly talk about my concept and its progress and secondly to test my prototype with an experienced tester of inclusive media – Stephan Heinke, who is blind himself. Eventhough it turned out (once again) that the device is not jet on level of development to be a useful navigation aid for blind people, the test and the conversation still led to a lot of productive feedback for me. Hopefully this won't be our last meeting, as the institute can in principle imagine to support the project in an adequate form. 

  • Stability Issues, Raspi 0 Disappointment

    Jakob Kilian05/06/2019 at 08:39 2 comments

    This is a short update of my project.
    End of this week I will have a short meeting / testing with a big german research organization – therefore I am working again on stability issues.
    The "royale" library from PMDtec crashes after some time so I build a mechanism that checks the activity of the library and reinitializes it after inactivity. Restart now also happens, when camera gets detached accidentally – the whole starting procedure furthermore just takes about 5 seconds now...
    Currently I am building a interface for android phones to check the status of the program. It is getting better and hopefully I am able to finally upload the code after this session. 

    Furthermore I tested the Raspi Zero W with my Code. Didn't know that it is just a ARMv6 processor, so it didn't work due to NEON commands off the "royale" library. Still looking for an powerful alternative smaller to RPi 3b+ to save energy and size. Thinking about ODROID C0 but it doesn't seem to be a widely used board... 

  • Intermediate: new case, new hardware assembly, new code

    Jakob Kilian02/16/2019 at 10:13 0 comments

    Having an upcoming TV shooting with a blind test person in mind, I reached another intermediate goal: as already mentioned in my previous post I moved the platform from laptop to Raspi 3b+ and from crappy plugged electronics to a soldered shield doing all the output stuff. Things can be much smaller in future being soldered on a pcb made for this purpose. But for now I used the breakout boards of the TCAs and the DRVs as it is easier to solder them and to replace them in case of faults.

    Power Supply

    I decided to use standard USB powerbank for power supply as they are interchangeable, they already come with housing, charge controller and some safety features. As my nine vibration actuators take around 400mA at 3V I decided to power them from the raspi’s 3V output pin. Therefore I can power the whole device with a  powerbank. As one read in many discussions on the internet, voltage is a critical thing for the 3b+. Using bad USB cables (and most of them out there are indeed bad) voltage can drop significantly under full load. That is because powerbanks are not made to keep the voltage on a specific level and those cables come with very thin wires (some of mine had more than 0.2 ohm on 20cm which would lead to -0.4v at 2A). As it is difficult to find cables that have specific data about their quality (like the AWG value) I just ordered a few to test. In the end I chose a 10cm cable from System-S and got a the Anker PowerCore Lite 10000mAh which comes with 5.2V output (little bit higher than others) and which is said not to have huge voltage drops. With this setting I have relatively stable voltage supply and didn’t get the voltage-warning-bolt.

    Range Adjustment

    As the use cases of the device differ a lot (e.g. indoor vs. outdoor) I wanted to have a adjustable depth range. As I needed an interface which is useable for blind people, I went for an analog potentiometer. It is input and output at the same time as it can be used both to adjust

    the value as well as to read the current range setting. Using an analog poti with a raspberry is more difficult than I thought, as you need a separate a/d converter because of the absence of analog inputs on the raspi. The Digispark microcontrollers are a good workaround for this (thanks to: Paul C. Brown) I had some trouble setting it up but in the end it worked out pretty well. For a better tangibleness I printed out an asymmetrical poti cap.

    Shield and Case

    The soldering of the shield and the mounting on the raspi wasn’t planned very well but included a lot of improvising. As I wrote in the beginning, this is just an intermediate step and the ICs can be placed in much higher density on a single pcb in future. When everything was fixed I lasercut an acrylic case to protect the electornics from shorts and pressure whilst ensuring enough air flow to protect the raspi from overheating. Everything was crammed in the cheap sport arm band you can see on the pictures to wear it at the upper arm. The camera on the glove also got a new 3d printed support:

    Code

    I am designer and therefore not an expert in coding. To port the code to the raspberry I I therefore got help from friends at Press Everykey. After loads of performance and stability issues I added the adjustable range function and the possibility to stream the raw tof camera data to a tablet as well as the processed tiles that control the vibration actuators. As I already announced, I will release the code, when it is on a clean and understandable level. If you are interested in hundreds of messy code lines you can of course contact me now… ;-)

    Shooting 

    After replacing a broken actuator in the last night, the stuff was finally ready for the shooting with the german tv station. Technically everything went well but due to a fix schedule there wasn’t much time to go into detail. The blind test person had problems to get familiar with the new device as we didn’t really have the possibility to train the new inputs...

    Read more »

  • Moving the Platform...

    Jakob Kilian02/08/2019 at 11:03 0 comments

    The first prototypes of this project started with a laptop in the backpack. This had many disadvantages like overheating, weight and a lack of comfort. Also the code was very buggy in the beginning, running into crashes that I couldn't reconstruct.

    So I started moving the project to a new platform (Raspi) and overworking the hardware and code. I am now at a much more portable and stable point and I will release more infos and hopefully the code (when I cleaned up my beginner-level writing) very soon.

    For now here are some progress photos:

View all 7 project logs

Enjoy this project?

Share

Discussions

Brandon wrote 5 days ago point

Hey just found this looking through the Hackaday Prize entries!  LOVE this idea.  

So we're working on a Depth sensing + AI platform (aptly named DepthAI) that allows object localization in real-time.  

Would be more than happy to send you one if you think it would be useful for this.  My first thought is that it would be... as it gives information on what objects are, and where they are in 3D space, real-time.

So then this can be used to map vibrations to the hand through your system.

We're also a hackaday project, here: https://hackaday.io/project/163679-luxonis-depthai

Thoughts?

And thanks for making such a cool system!  So we once did something similar for radio waves... so you could search out transmitters through vibrations on your hand.  And also the Commute Guardian bike safety system, which is why we're making DepthAI, it's our end goal, will use haptic feedback to the biker as well as a warning.  It's for giving 'eyes in the back of your head' for bike commuters.

Cheers,
Brandon & The Luxonis Team!  

  Are you sure? yes | no

Jakob Kilian wrote 5 days ago point

Hi Brandon,

thanks for your nice words! 

Just checked your project, it is amazing and super sophisticated. I always thought about something like this to add to my very basic orientation device. But I did not have enough engineering skills to do so… So you are absolutely right, this would be a great combination and I’d be extremely happy to receive on of the shields or work together with you!

I am super busy right know, due to work, other stuff and also (luckily) due to the Hackaday Price. I’ll send you a PM so that we can stay in contact and have a chat latest in october.

Thank you in advance. And also props for your great project – we live in a very interesting time.

Jakob

  Are you sure? yes | no

Brandon wrote 4 days ago point

Hey thanks for all the kind words!  And yes definitely will keep in touch.  Will keep you posted on hardware and firmware progress.  

Thanks again,

Brandon

  Are you sure? yes | no

Jakob Kilian wrote 02/15/2019 at 12:53 point

thanks guys. Just wanted to add: the basic idea behind that is kind of old – check out Paul Bach-y-Ritas amazing studies back in the 1960s (https://youtu.be/7s1VAVcM8s8?t=147) and loads of others (keyword sensory substitution). New is the 3D tof technology and cheap hardware in general. This brings everything on a new level...

  Are you sure? yes | no

Josh Cole wrote 02/15/2019 at 06:38 point

This is such a cool idea! I've been thinking recently about how to use other sensory mechanisms beyond eye sight in order to convey information. This is next level though, and could be life changing for some people. I can't wait to see/feel where it goes next!

  Are you sure? yes | no

Malek wrote 02/12/2019 at 19:24 point

great work keep going

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates