Close
0%
0%

Unfolding Space

see with your hands

Public Chat
Similar projects worth following
The “Unfolding Space” glove is a prototypical open source navigation aid system for the visually impaired.

The project deals with sensory substitution, a phenomenon by which the function of one missing or dysfunctional sensory modality is replaced (substituted) by stimulating another one. The outcome is a prototype that projects a 3D picture, generated by a special camera, as vibration patterns on the back of the hand. Visually impaired people can therefore use their tactile modality of the hand to actually see.

This Project started back in 2018 and was initiated as my BA thesis project. I published everything open source to inspire others to work on that topic. You can find all the files, instructions, codes, papers and materials linked on this project page.

As I'm an Interaction Designer, this mainly is a design project. Regarding soft- and hardware development I'm still learning and I am sorry if some things are not in line with other well ordered projects. Don't hesitate to drop me some lines, if you have any question or feedback!



Basic Functionality

A 3D generated picture from a depth camera is haptically projected on the back of the hand by using vibration patterns. The location of a vibration depicts an object’s relative position in space, the strength of the vibration represents its distance.

Scientific Background

The theoretical background of this project is called Sensory Substitution. A phenomenon by which the function of one missing or faulty sensory modality is replaced (substituted) by stimulating another one – in this case the tactile modality. At the beginning of the substitution process, this new stimulation has to be actively interpreted by analysing the tactile stimulus. But after some training the new visual-like Input becomes implicit and gets processed subconsciously. Users begin to see the space in front of them.

Motivation

These scientific efforts started more than 50 years ago, but even today there are almost no blind people using substitution processes to handle the absence of their visual system. All attempts to develop a device for the broad market failed.

Process

Many projects failed on a practical implementation. An analysis revealed that despite the elaborate technology used by them, design and usability issues were often not taken into place. Therefore I followed an open and iterative rapid-prototyping approach to quickly work out strengths and potentials and to identify limitations of the hardware and algorithms. Even though my theoretical work about Sensory Plasiticity had already predicted many aspects, it was this process of prototyping that lead me to a functioning device that quick.

Here is my old documentation:

General_Shematic-V02.pdf

A schematic of the basic configuration of the system. There is a separate diagram for the motor board PCB driving the vibration motors.

Adobe Portable Document Format - 2.65 MB - 11/09/2019 at 18:34

Preview
Download

Computing_Unit_Case-V01.stl

This is the STL export of the custom-made case. Go to https://a360.co/2ojlpKu to see the fusion file in the browser and download it in various file formats. After the case is printed follow the instructions for assembly.

Standard Tesselated Geometry - 635.92 kB - 11/09/2019 at 16:02

Download

Motor_Board_V02.zip

PCB Design for the "Motor Board V0.2" including: - Eagle File - Gerber File - Aisler.net project (https://aisler.net/p/DMKOGATR) ready to order incl. parts Check the instructions for more information

Zip Archive - 395.44 kB - 11/09/2019 at 16:00

Download

Motor_Board_V02-Schematic.png

Schematic diagram of the Motor_Board

Portable Network Graphics (PNG) - 220.79 kB - 11/09/2019 at 16:00

Preview
Download

Parts_List_V02.pdf

PDF Parts list with links and prices

Adobe Portable Document Format - 76.58 kB - 11/09/2019 at 16:00

Preview
Download

View all 8 files

  • 1 × PMDtechnologies Pico Flexx 300€
  • 1 × Custom Made PCB ca. 50€
  • 9 × DRV2605l included in "custom made PCB"
  • 2 × TCA9548A included in "custom made PCB"
  • 9 × G0832012 3€

View all 13 components

  • My project won a Hackaday Prize Honorable Mention!

    Jakob Kilian3 days ago 3 comments

    The Hackaday Prize winners are out! And Unfolding Space won an honorable mention for "Best Concept".

    Once more I can express my admiration for the high level of quality and elaboration of the finalists' projects. I am super happy and I really feel "honored" (haha) to be among this year's winners.

    I definitely know where to spend money on and I am very happy to have this opportunity and privilege to continue to work on something that exciting and demanding at the same time.

    Thank for your support and also thanks to Supplyframe and the other Sponsors to make this happen.

    Cheers!

  • Updated Files and Schematics

    Jakob Kilian11/09/2019 at 18:42 0 comments

    I invested some time to make my files, schematics and instructions clearer. You can see the updated picture all over the page. Comment, if you spotted a mistake! This is my new general schematic (would you call it the same? Don't know if it is the right term...)

  • Last but not least: My Video Submission for the Hackaday Prize

    Jakob Kilian10/01/2019 at 13:47 1 comment

    An finally there is my submission video for the Hackaday Prize 2019.

    As always: feedback is very welcome!

  • Instructions Online Now

    Jakob Kilian10/01/2019 at 00:53 0 comments

    In another night shift I uploaded some Instructions. Certainly this won't be the last word, things are changing all the time and I am still about to figure out the best way to build the prototype... 

    But eh, now I have some fancy visualizations and you maybe can understand a little better what I did here ;-) Have fun with it!

  • Uploaded the Code – Finally

    Jakob Kilian09/29/2019 at 10:52 0 comments

    Hey folks, it took me some effort but finally I uploaded my code on Github! :-)

    All the amateurish coding skills I have I taught myself. I am getting better, but what I just start to learn, is how to structure code and how to document it in a way, that others – maybe contributors – can understand it properly and fast... This upload may be a starting point, but I also know, that there is a lot more to do. 
    Hope you can find your way through! 

  • Connection Issues - Solved?

    Jakob Kilian09/28/2019 at 17:16 0 comments

    Another short Log on the busy way of reviewing and documenting all my work. (Next Log will be about the code, promised) 

    I wish to have just one cable and therefore one plug to connect between computing unit and glove. But as I need 8 wires (2 for power consumption for the motors, 2 for i2c and another 4 for USB 2.0 fro the ToF camera) it was difficult to find a standard with a handy, small plug and socket... Therefore I just used a HDMI micro cable – Neat idea isn't it? Nope!. It seemed to work in the beginning and I could successfully connect the camera and drive the motors. 

    BUT...

    After some time the trouble began: the camera disconnected randomly and finally wasn't connectable at all. A friend of me now told me the (possible) reason for that: Even on this short distance of 40cm the HDMI cable might cause problems as it doesn't have USB specs and therefore a different impedance. I would never had thought of this, but after some research of course it makes sense.

    That means: For the development I am currently connecting the camera via an extra USB cable (meeeh), but I am looking for an alternative cable...

    Stay tuned,

    Jakob

  • Casing – the second attempt

    Jakob Kilian09/24/2019 at 08:57 0 comments

    Over the summer it became quite silent around me and the project. Now – with a little bit more time – I'd like to recap my last steps. Starting with the casing:

    As I noted in my previous casing post, I had problems with the first case and the velcro fastener. It couldn't fasten the computing unit tight enough, was difficult to apply without help and it couldn't fulfill the demands for comfort to be a everyday-worn gadget. Therefore I repeated the process, I did research and in the end I came up with the following solution:

    As a basis I used a15$  detachable phone armband. It comes with a solid band with velcro fastener and a click-attachment. I designed the case so that it can receive the plastic connector, as you can see here:


    This new design totally fulfilled my expectations and is a good basis for upcoming experiments with blindfolded and blind subjects. Furthermore the size cannot be significantly reduced further as the main part is still the battery which already has minium dimensions for the 10000mAh it stores...

    I am currently working hard to provide further plans, details and code in the next days. Stay tuned and thanks for your support!

  • I Reached Hackaday Price Finals - Yeah!

    Jakob Kilian09/18/2019 at 12:40 0 comments

    Hey together,

    this will be a short log, as I am a bit busy and as I'm working hard on the preparation for the Hackaday Price Finals 2019 which I have been nominated for. In the final round there are 20 amazing projects – all of them have huuuge potential and some of them are indeed very advanced and technically complex. Therefore I feel very honoured to be nominated with a more interaction-, design- and concept-focused project. 

    Furthermore I won a Mark in the Taiwanese Golden Pin Concept Design Award and I run for the "best ofs". In this connection I got also featured on designboom.com which I am extremely happy too. :-)

    Last but not least I wrote an application for a research assignment together with my university. If this will work, I could dive deeper into the research and hopefully speed up the process..

    These are the reasons why I am currently producing video footage for the documentation like this interview here ;-)

    There is still a lot to do, but there will be news very soon. Thanks for your support!

    Jakob

  • Raspi Slimdown, New Case

    Jakob Kilian06/16/2019 at 16:04 0 comments

    Short update on my progress:

    To make everything smaller and more mobile, I designed a new case with a ultra slim 10000mah powerbank from TNTOR. As I didn't need the USB and Ethernet sockets I slimed down my Raspi as you can see on the pictures. The new casing is now already quite neat and slim but the elastic band with velcro fastening didn't work out as expected. It is a little bit too limp and has too less adjustability to different arm. I might have to come up with a different solution here. 

  • Intermediate: PCB for driving the 9 vibration actuators

    Jakob Kilian06/08/2019 at 11:38 0 comments

    The bulky self made board out of 9 DRV2605l breakouts can no go into retirement. I successfully planned, ordered and equipped PCB is finally there and – hoooray – it works.

    It took me some efforts to learn how to design a proper circuits and even more to reflow the tiny parts (thanks to my friends at Presseverykey). But it seems like I succeeded. Most of you may be familiar with circuit boards – to me at least it was a great excitement to see it working in the end ;-)

    Next Step: As I couldn't find a worthy slim/small alternative single chip computer I decided to dismantle all the ports of the Rpi3b+ to finally fit everything except the battery directly IN the glove. Lets see how this works...

View all 14 project logs

  • 1
    Download and View the Material Carefully

    Building and assembling the prototype entails considerable effort. But generally it should be feasable to hobbyists and makers as well as to professionals in electronics and hardware development. There is no doubt that many steps will differ from build to build as the used materials and the tools may differ. This guide therefore tries so give you an overview over decisions I made and the strategy I followed, when I build my unit. As I am very interested in your way and also in your questions and proposals for improvement: Comment here or drop me a line, when you start to work on this!



    Have fun with this guide, I hope it will help!



  • 2
    COMPUTING UNIT: Finf, Buy and Print Components

    Plenty of stuff is needed and some parts may need some improvisation as you maybe can't purchase them all over the world. 

    We're starting with the casing for the computing unit. Therefore you need:

    The Armband and the Adapter
    There are many offers out there for detachable sport armbands. You can find them in different variants. This one and this one should come closest to the one I used. You can also find the parts separate, like this adapter

    TNTOR Power Bank
    Might be difficult to find outside of Europe – at least I couldn't do so. It is a very thin (8.8mm) power bank which in general keeps it promises.

    Raspberry Pi 3b+
    You will best know where to purchase. I didn't test many other versions, but performance should be best on the newest... Raspi ZERO DOESN'T WORK! As is doesn't know NEON commands, used by the camera's library.

    The 3D printed Case itself
    You can find the 3D files in the files section. Hope you have access to a printer. Double check the measurements if your components fit!





  • 3
    COMPUTING UNIT: Slim Down the Raspberry

    To slim the computing unit down, I removed the USB ports, the Ethernet port and the "Display" socket and cut the end of the Raspberry Pi 3b+. Have a look at this and this tutorial!
    First the tricky part: unsolder and clean up the soldering joints. When the parts are removed, cut the raspberry along the green line: 


    Thanks zu Killar on Sudomod.com

    After the diet mine looked like this:

View all 9 instructions

Enjoy this project?

Share

Discussions

Michał wrote 10/13/2019 at 11:38 point

Well done! Great work.

Why you did not use opposite side of hand? It seems to be more natural, more sensitive, more receptors are there...

Go further... Scale up effectors - put wider matrix on clothes, shoes. So you get much better, continous, general, directional orientation in space, regardless of moving parts of system like gloves. Of course, vibration modules are nice but still bulky, so maybe try delicate(!) HV, electroactive polymer (shape changing fabric) or piezo something...😸 If this method may be practical it should be as printable/flexible as possible, fluently integrated with clothes. A big challenge but it is possible...

  Are you sure? yes | no

Jakob Kilian wrote 11/09/2019 at 18:45 point

Hey Michal, sorry for the late reply. Somehow I oversaw your post...

Actually I choose to use this side, because I wanted to maintain the full functionality of the hand. Using the vibration motors, that couldn't be guaranteed on the palm. Furthermore I wanted the sensation as easy to learn as possible and during my trials I found out that it would be easier to have the sensation on the side where the sensed object actually is located, which is the back of the hand if the hand is in a natural position.

Yes, that would be an interesting step to do research on! Still the hand has one big advantage: the degree of movement. In cognitive science it is believed that (new) sensory input can only be learned by movement (and relating feedback) – the more movement the faster the learning. In my first experiments this theory has somehow been confirmed: you could even shake your hand in various directions but you still get the relation to the real world around. 

But what I definitely want to do implement ist a sensor system on the ground level to identify small obstacles lying on the ground. This would be a great opportunity to test other parts of the body...

And last but not least (sorry for the long answer) I am working on electro stimulation. This could be a small, lightweight, scaleable and low-power alternative to the motors. I just wanted to start with a more easy way (motors) before I grill peoples hand with high voltages (which you need for stimulation, haha). 

What is HV? High Voltage? 

Thanks for the tips, I keep them in mind!

  Are you sure? yes | no

Brandon wrote 09/17/2019 at 20:06 point

Hey just found this looking through the Hackaday Prize entries!  LOVE this idea.  

So we're working on a Depth sensing + AI platform (aptly named DepthAI) that allows object localization in real-time.  

Would be more than happy to send you one if you think it would be useful for this.  My first thought is that it would be... as it gives information on what objects are, and where they are in 3D space, real-time.

So then this can be used to map vibrations to the hand through your system.

We're also a hackaday project, here: https://hackaday.io/project/163679-luxonis-depthai

Thoughts?

And thanks for making such a cool system!  So we once did something similar for radio waves... so you could search out transmitters through vibrations on your hand.  And also the Commute Guardian bike safety system, which is why we're making DepthAI, it's our end goal, will use haptic feedback to the biker as well as a warning.  It's for giving 'eyes in the back of your head' for bike commuters.

Cheers,
Brandon & The Luxonis Team!  

  Are you sure? yes | no

Jakob Kilian wrote 09/18/2019 at 12:23 point

Hi Brandon,

thanks for your nice words! 

Just checked your project, it is amazing and super sophisticated. I always thought about something like this to add to my very basic orientation device. But I did not have enough engineering skills to do so… So you are absolutely right, this would be a great combination and I’d be extremely happy to receive on of the shields or work together with you!

I am super busy right know, due to work, other stuff and also (luckily) due to the Hackaday Price. I’ll send you a PM so that we can stay in contact and have a chat latest in october.

Thank you in advance. And also props for your great project – we live in a very interesting time.

Jakob

  Are you sure? yes | no

Brandon wrote 09/18/2019 at 19:21 point

Hey thanks for all the kind words!  And yes definitely will keep in touch.  Will keep you posted on hardware and firmware progress.  

Thanks again,

Brandon

  Are you sure? yes | no

Jakob Kilian wrote 02/15/2019 at 12:53 point

thanks guys. Just wanted to add: the basic idea behind that is kind of old – check out Paul Bach-y-Ritas amazing studies back in the 1960s (https://youtu.be/7s1VAVcM8s8?t=147) and loads of others (keyword sensory substitution). New is the 3D tof technology and cheap hardware in general. This brings everything on a new level...

  Are you sure? yes | no

Josh Cole wrote 02/15/2019 at 06:38 point

This is such a cool idea! I've been thinking recently about how to use other sensory mechanisms beyond eye sight in order to convey information. This is next level though, and could be life changing for some people. I can't wait to see/feel where it goes next!

  Are you sure? yes | no

Malek wrote 02/12/2019 at 19:24 point

great work keep going

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates