Open Source Sensory Substitution Device

The Unfolding Space Glove allows blind or visually impaired users to haptically sense the depth of their surrounding space and thus (hopefully) better navigate through it. The device was drafted and developed within the framework of a four-year design research project. In 2021, the prototype was tested in an empirical study with 14 sighted and blind subjects with the results being published. 

Substituting missing visual stimuli by means of another sensory modality is referred to as Sensory Substitution in scientific terminology. The code and all components of the project have been released to the public under an Open Source licence. Depending on the field and audience, however, the device can have other labels as well: 

Sensory Substitution Device

There is a tradition of more than 50 years of research on Sensory Substitution. It is related to recent findings in neuroplasticity and cognitive science that describe the brain's ability to adapt to changes in the sensory periphery even in adulthood.

Navigation Aid System

Rather than replacing the full depth of visual information, the purpose of the device is to aid blind or visually impaired people in navigating independently and effectively. Object recognition is thus deliberately not included in the functional spectrum.

Visual Prosthesis

The two aforementioned aspects also make it, at least partially, a Visual Prosthesis. Users of similar sensory substitution devices report vision-like experiences after some training - the corresponding brain activity can be observed in MRI scans.

Current Prototype (2021)

The worn glove in close-up in front of the user's body. The components such as the camera, fan and circuit board are clearly visible.
Closeup of the glove
Frontal view of a user wearing the glove and extending his arm forward as if searching for an object.
Frontal view of user with glove in basic posture
User has a grey USB power bank in his hand, which he attaches to an upper arm band.
Attaching the power bank
The user plugs a USB cable into the power bank, which is now attached to the upper arm.
Plugging the Cable into the Power Bank
The user now plugs the USB cable into the power socket on the glove. A close-up of the glove is shown.
Plugging in power connection at the glove
The board is just attached to the glove. Velcro strips are visible on both sides, which are used for fastening.
Velcro fastening the PCB
Close-up of the circuit board without glove against a white background. Many technical components attached to the circuit board can be seen in detail. Among them: The black camera, the black fan, connectors and other electronic components.
Closeup of the Unfolding Space Carrier Board
A user holds the arm with glove in front of the body in the basic posture and apparently makes an effort to interpret what he feels.In the background, an obstacle course made of cardboard.
Using the device in an obstacle course
A user wears the glove on his left hand and reaches with his right, free hand for an object standing directly in front of him. In the background, the obstacle course consisting of cardboard can be seen again.
Grabbing for an obstacle in the training
Previous
Next

Previous Prototypes (2018-2020)

Frontal View of the Prototype worn by a user in front of a wall
2018 | The prototype worn by a user
A close up shot of the vibration motors used to transmit the data to the skin
2018| Glove in close-up
A blindfolded person is wearing the prototype on a footpath
2018 | Blindfolded person testing the device
A person using the prototype to navigate through an underground train station
2018 | The glove in an underground train station
An early version of the prototype with lots of cables hanging around
2018 | First functional test in rapid prototyping iterations
Previous
Next

Background Information

Basic Functionality

A depth image from a 3d camera is haptically projected on the back of the hand by using vibration motors. The location of a vibration depicts an object’s relative position in space, the strength of the vibration represents its distance. See the demo video on the right for better understanding.

Project Background

There is a multitude of projects that have dealt with the topic of Sensory Substitution, but to date there are only very few practical implementations of the idea, which in turn are used by a negligible small number of people. While extremely sophisticated technology is used, design and user-friendliness often suffer. I therefore started researching the topic in an open and iterative rapid prototyping process. Problems of existing solutions should be included, the operation and learning should be easier and a higher acceptance for the system should be created by addressing usability and interaction design requirements.

Current Prototype & Empiric Study in 2021

Over the years, several prototype versions were created, up to the current version presented here. In the summer of 2021, I was able to test it in an empirical study with 14 blind and sighted test persons as part of my master’s thesis. As expected, it turned out that the path to a practical aid for blind people is still long – nevertheless, the study revealed a number of aspects that the Unfolding Space Glove already meets or which are valuable for further developments. Also see section “Publication” for further Details and a link to the paper.

Early protoype from 2018 with technical equipment in a backpack
Frontal View of the Glove worn by a user with labels on the three main components: the glove, a power connection cable and the power bank at the upper arm
Close-up of the circuit board without glove against a white background with labelling of the components: An IDC connector as a connection to the glove, a Time of Flight 3D camera, a Rasperry Pi Compute Module 4, a fan to cool the Compute Module and, above all, the custom-made PCB as a basis.

Technical Details

The Prototype now consists of only three components: a 5V USB power bank, a USB-C power cable and the glove itself hosting a mini computer and the 3D camera.

Once those three are connected you are ready to go:

  • External hardware or specific premises are no longer required.
  • Runs indoor and outdoor.
  • No need specific lighting conditions.
  • A single battery charge holds around eight hours.

Total component costs are about $ 500.  For an overview, see the pictures on the left. For details, parts lists and building instructions, please go to the GitHub repo linked below.

Future of the Project

After four years of active development, I will let this project rest due to lack of time and money. Of course, questions, tips or feedback are always welcome and maybe the project will continue with or without me at another time. That would make me really happy – the foundation stone has been laid, all the data is open. 

Publications

As mentioned I conducted a study about the Unfolding Space Glove in summer 2021. A paper about this study is expected to be published soon. You can already read the abstract below:

This paper documents the design, implementation and evaluation of the Unfolding Space Glove—an open source sensory substitution device. It transmits the relative position and distance of nearby objects as vibratory stimuli to the back of the hand and thus enables blind people to haptically explore the depth of their surrounding space, assisting with navigation tasks such as object recognition and wayfinding. The prototype requires no external hardware, is highly portable, operates in all lighting conditions, and provides continuous and immediate feedback—all while being visually unobtrusive. Both blind (n = 8) and blindfolded sighted participants (n = 6) completed structured training and obstacle courses with both the prototype and a white long cane to allow performance comparisons to be drawn between them. The subjects quickly learned how to use the glove and successfully completed all of the trials, though still being slower with it than with the cane. Qualitative interviews revealed a high level of usability and user experience. Overall, the results indicate the general processability of spatial information through sensory substitution using haptic, vibrotactile interfaces. Further research would be required to evaluate the prototype’s capabilities after extensive training and to derive a fully functional navigation aid from its features.

MDPI – Sensors | Scientific Paper

Published on 26.02.22 in the special issue "Spatial Perception and Navigation in the Absence of Vision" of MDPI – Sensors.

ResearchGate | All Publications

My profile on ResearchGate hosting all publications. 2018 bachelor thesis and proposal (theoretical paper) are only available in German.

Code and Content

Creative Commons LicenseContent of this website is licensed under a Creative Commons Attribution 4.0 International License.

In order to inspire other designers, developers or commercial providers to work within this field and to make Sensory Substitution available to as many people as possible this project is Open Source. All contents are under CC-BY-4.0 license, the code is under MIT license. Please check the LICENSE files in the respective repositories. This website hosts general information about the project. Please find all components, files, documentation and publications related to this project at the locations linked below:

GitHub | Main & Code

This is the main repository of the project. Besides general information, software code and documentation on project setup, installation and use can be found here.

GitHub | Hardware

Building instructions for the glove and the Unfolding Space Carrier Board, PCB files, parts lists / BOMs, 3D design files and more...

GitHub | Monitor

Multi platform project in Unity 3D of the Monitoring App that allows to monitor the activity of the device via Wifi and adjust settings. Including builds and documentation.

Vimeo | Study Videos

Collection of exemplary videos from the study in 2021 showing: pocedure of the trials, examples of efficient wayfinding with the glove, comparisons between white long cane and glove and more ...

Hackaday | Instructions & Blog

Project page on hackaday.io, where you can find building instructions, I regularly post news and you can discuss about the project.

Awards

Winner of the "Kölner Design Preis / Toby E. Rodes Award" 2018

Schriftlogo des Kölner Design Preises 2018 auf weißem Hintergrund
HackadayPrize 2019 - Honorable Mention (Best Concept)
Hackaday Prize Schriftlogo

Golden Pin Concept Design Award 2019 - Design Mark Winner

Schriftlogo des Golden Pin Concept Design Awards auf weißem Hintergrund

Lucky Strike Junior Designer Award 2018 – Special Recognition

Schriftlogo des des Lucky Strike Junior Designer Awards 2018

Press Releases

In English:
Other Languages:

Credits

The project wouldn’t have been possible without the help of many people, to whom I would like to express a big thank you at this point:

  •  Kjell Wistoff for his active support in setting up, dismantling and rebuilding the study room, organising the documents and documenting the study photographically.

  • Trainer Regina Beschta for a free introductory O&M course and the loan of the study long cane.

  • Tim Becker and Matthias Krauß from Press Every Key for their open ear when giving advice on software and hardware.

  • Köln International School of Design/KISD (TH Köln) and the responsible parties for making the premises available over this long period of time.

  • Tom Bieling, Ulrike Gollner & Gesche Joost 

And all those who provided guidance in the development of the prototype over the past years and now in the implementation and evaluation of the study. Specifically (and probably more that I can not remember at this point):

Jannik Nitz, Connor Shafran, Johanna Warchol, Marco Reitano, Brandon Gilles, Niklas Isselburg, Eduard Paal, Dmitry Kuznetsov, Sebastian Miller, Sören Stirn, Ronnie Balcazar

Hardware sponsoring was granted by:

pmdtechnologies text logo

pmdtechnologies

Providing a Pico Flexx 3D camera.

Munitec text logo

Munitec GmbH

Providing multiple glove samples that made into the final prototype.

Contact me

I am currently working as research assistant at Köln International School of Design (TH Köln).

Feel free to contact me via Email, via Hackaday.io, via Github.com or visit my portfolio page.

Jakob Kilian
Interaction Designer
Skip to content