Open Source Sensory Substitution Device

The Unfolding Space Glove allows blind or visually impaired users to haptically sense the depth of their surrounding space and thus (hopefully) better navigate through it. The device was drafted and developed within the framework of a four-year design research project. In 2021, the prototype was tested in an empirical study with 14 sighted and blind subjects with the results being published. 

Substituting missing visual stimuli by means of another sensory modality is referred to as Sensory Substitution in scientific terminology. The code and all components of the project have been released to the public under an Open Source licence. Depending on the field and audience, however, the device can have other labels as well: 

Sensory Substitution Device

There is a tradition of more than 50 years of research on Sensory Substitution. It is related to recent findings in neuroplasticity and cognitive science that describe the brain's ability to adapt to changes in the sensory periphery even in adulthood.

Navigation Aid System

Rather than replacing the full depth of visual information, the purpose of the device is to aid blind or visually impaired people in navigating independently and effectively. Object recognition is thus deliberately not included in the functional spectrum.

Visual Prosthesis

The two aforementioned aspects also make it, at least partially, a Visual Prosthesis. Users of similar sensory substitution devices report vision-like experiences after some training - the corresponding brain activity can be observed in MRI scans.

Current Prototype (2021)

Previous Prototypes (2018-2020)

Background Information

Basic Functionality

A depth image from a 3d camera is haptically projected on the back of the hand by using vibration motors. The location of a vibration depicts an object’s relative position in space, the strength of the vibration represents its distance. See the demo video on the right for better understanding.

Project Background

There is a multitude of projects that have dealt with the topic of Sensory Substitution, but to date there are only very few practical implementations of the idea, which in turn are used by a negligible small number of people. While extremely sophisticated technology is used, design and user-friendliness often suffer. I therefore started researching the topic in an open and iterative rapid prototyping process. Problems of existing solutions should be included, the operation and learning should be easier and a higher acceptance for the system should be created by addressing usability and interaction design requirements.

Current Prototype & Empiric Study in 2021

Over the years, several prototype versions were created, up to the current version presented here. In the summer of 2021, I was able to test it in an empirical study with 14 blind and sighted test persons as part of my master’s thesis. As expected, it turned out that the path to a practical aid for blind people is still long – nevertheless, the study revealed a number of aspects that the Unfolding Space Glove already meets or which are valuable for further developments. Also see section “Publication” for further Details and a link to the paper.

Early protoype from 2018 with technical equipment in a backpack

Technical Details

The Prototype now consists of only three components: a 5V USB power bank, a USB-C power cable and the glove itself hosting a mini computer and the 3D camera.

Once those three are connected you are ready to go:

  • External hardware or specific premises are no longer required.
  • Runs indoor and outdoor.
  • No need specific lighting conditions.
  • A single battery charge holds around eight hours.

Total component costs are about $ 500.  For an overview, see the pictures on the left. For details, parts lists and building instructions, please go to the GitHub repo linked below.

Future of the Project

After four years of active development, I will let this project rest due to lack of time and money. Of course, questions, tips or feedback are always welcome and maybe the project will continue with or without me at another time. That would make me really happy – the foundation stone has been laid, all the data is open. 

Publications

As mentioned I conducted a study about the Unfolding Space Glove in summer 2021. A paper about this study is expected to be published soon. You can already read the abstract below:

This paper documents the design, implementation and evaluation of the Unfolding Space Glove—an open source sensory substitution device. It transmits the relative position and distance of nearby objects as vibratory stimuli to the back of the hand and thus enables blind people to haptically explore the depth of their surrounding space, assisting with navigation tasks such as object recognition and wayfinding. The prototype requires no external hardware, is highly portable, operates in all lighting conditions, and provides continuous and immediate feedback—all while being visually unobtrusive. Both blind (n = 8) and blindfolded sighted participants (n = 6) completed structured training and obstacle courses with both the prototype and a white long cane to allow performance comparisons to be drawn between them. The subjects quickly learned how to use the glove and successfully completed all of the trials, though still being slower with it than with the cane. Qualitative interviews revealed a high level of usability and user experience. Overall, the results indicate the general processability of spatial information through sensory substitution using haptic, vibrotactile interfaces. Further research would be required to evaluate the prototype’s capabilities after extensive training and to derive a fully functional navigation aid from its features.

MDPI – Sensors | Scientific Paper

Published on 26.02.22 in the special issue "Spatial Perception and Navigation in the Absence of Vision" of MDPI – Sensors.

ResearchGate | All Publications

My profile on ResearchGate hosting all publications. 2018 bachelor thesis and proposal (theoretical paper) are only available in German.

Code and Content

Creative Commons LicenseContent of this website is licensed under a Creative Commons Attribution 4.0 International License.

In order to inspire other designers, developers or commercial providers to work within this field and to make Sensory Substitution available to as many people as possible this project is Open Source. All contents are under CC-BY-4.0 license, the code is under MIT license. Please check the LICENSE files in the respective repositories. This website hosts general information about the project. Please find all components, files, documentation and publications related to this project at the locations linked below:

GitHub | Main & Code

This is the main repository of the project. Besides general information, software code and documentation on project setup, installation and use can be found here.

GitHub | Hardware

Building instructions for the glove and the Unfolding Space Carrier Board, PCB files, parts lists / BOMs, 3D design files and more...

GitHub | Monitor

Multi platform project in Unity 3D of the Monitoring App that allows to monitor the activity of the device via Wifi and adjust settings. Including builds and documentation.

Vimeo | Study Videos

Collection of exemplary videos from the study in 2021 showing: pocedure of the trials, examples of efficient wayfinding with the glove, comparisons between white long cane and glove and more ...

Hackaday | Instructions & Blog

Project page on hackaday.io, where you can find building instructions, I regularly post news and you can discuss about the project.

Awards

Winner of the "Kölner Design Preis / Toby E. Rodes Award" 2018

HackadayPrize 2019 - Honorable Mention (Best Concept)

Golden Pin Concept Design Award 2019 - Design Mark Winner

Lucky Strike Junior Designer Award 2018 – Special Recognition

Press Releases

In English:
Other Languages:

Credits

The project wouldn’t have been possible without the help of many people, to whom I would like to express a big thank you at this point:

  •  Kjell Wistoff for his active support in setting up, dismantling and rebuilding the study room, organising the documents and documenting the study photographically.

  • Trainer Regina Beschta for a free introductory O&M course and the loan of the study long cane.

  • Tim Becker and Matthias Krauß from Press Every Key for their open ear when giving advice on software and hardware.

  • Köln International School of Design/KISD (TH Köln) and the responsible parties for making the premises available over this long period of time.

And all those who provided guidance in the development of the prototype over the past years and now in the implementation and evaluation of the study. Specifically (and probably more that I can not remember at this point):

Jannik Nitz, Connor Shafran, Johanna Warchol, Marco Reitano, Brandon Gilles, Niklas Isselburg, Eduard Paal, Dmitry Kuznetsov, Sebastian Miller, Sören Stirn, Ronnie Balcazar

Hardware sponsoring was granted by:

pmdtechnologies text logo

pmdtechnologies

Providing a Pico Flexx 3D camera.

Munitec text logo

Munitec GmbH

Providing multiple glove samples that made into the final prototype.

Contact me

I am currently working as research assistant at Köln International School of Design (TH Köln).

Feel free to contact me via Email, via Hackaday.io, via Github.com or visit my portfolio page.

Der Autor und Macher des Projekts Jakob Kilian im Porträt
Jakob Kilian
Interaction Designer
Skip to content