Open Source Sensory Substitution Device

The “Unfolding Space” project deals with sensory substitution: A phenomenon by which the function of one missing or dysfunctional sensory modality is replaced (substituted) by stimulating another one. The outcome is a prototype that  projects a 3D picture, generated by a special camera, as vibration patterns on the back of the hand.

The blind can therefore use their tactile modality of the hand to actually see.

Science

This project started as bachelor thesis. It is therefore based on scientific research and can support new findings in the area of cognitive processes.

Design

The prototype is easy to use and learn as a result of using methods of interaction design and rapid prototyping.

Open Source

The project is released under CC-BY licence.
Therefore, the usage and further development of this material is not only allowed but encouraged!

Low Price

Due to production costs of less than 500 Euro it is cheaper than most of the available sensory substitution devices.

Video Teaser

Information

Basic Functionality

A 3D generated picture from a depth camera is haptically projected on the back of the hand by using vibration patterns. The location of a vibration depicts an object’s relative position in space, the strength of the vibration represents its distance.

Scientific Background

The theoretical background of this project is called Sensory Substitution. A phenomenon by which the function of one missing or faulty sensory modality is replaced (substituted) by stimulating another one – in this case the tactile modality. At the beginning of the substitution process, this new stimulation has to be actively interpreted by analysing the tactile stimulus. But after some training the new visual-like Input becomes implicit and gets processed subconsciously. Users begin to see the space in front of them.

Motivation

These scientific efforts started more than 50 years ago, but even today there are almost no blind people using substitution processes to handle the absence of their visual system. All attempts to develop a device for the broad market failed.

Process

Many projects failed on a practical implementation. An analysis revealed that despite the elaborate technology used by them, design and usability issues were often not taken into place. Therefore I followed an open and iterative rapid-prototyping approach to quickly work out strengths and potentials and to identify limitations of the hardware and algorithms. Even though my theoretical work about Sensory Plasiticity had already predicted many aspects, it was this process of prototyping that lead me to a functioning device that quick.

Awards and Nominations

Winner of the "Kölner Design Preis / Toby E. Rodes Award"

In 2018 the „Kölner Design Preis“ was awarded for the 11th time. For the first time it was commended by the Neumann-Hug SA – the preserver of the Toby E. Rodes estate. The awards go to final papers of the cologne design schools and are very much desired under the gradates.

Only in German – Sorry

„Mit ‚Unfolding Space‘ zeichnet die Jury ein Projekt aus, das hinsichtlich Idee, Entwurf und Umsetzung vorbildlich und bestechend ist. Jakob Kilian präsentiert mit dem Handschuh nicht nur einen voll funktionsfähigen Prototypen, sondern zeigt auch auf, wie die Kosten für ein solches Produkt zukünftig um ein Vielfaches reduziert werden können“

For more information go to the Kölner Design Preis website

"Special Recognition" at the Lucky Strike Junior Designer Award 2018

“With the Lucky Strike Junior Designer Award the Raymond Loewy Foundation is fostering the new generation of designers in Germany since 1992. With this award that comes with 10,000 Euros, the Foundation significantly contributes to design training. The Foundation’s continuous efforts contribute to raising the performance and quality levels at German design universities, support transparency of design training and develop guidelines for university design courses that bring results. Business and industry alike consider receiving an award as part of the Lucky Strike Junior Designer Award a ticket to the design profession. The award provides the new generation of designers a public forum to build networks in the design industry. This ensures the recognition of the broad as well as the industry-specific public.

The award cerenomy took place at saturday, the 24th of November 2018 at “stilwerk”, Hamburg. Together with 20 other young Designers I received a “special recognition” for the project. More information on the Lucky Strike Awards website.

Press Releases

Material

The whole project is published under CC-BY license (attribution) in order to inspire other designers, developers or commercial providers to work within this field and to make Sensory Substitution available to as many people as possible. In this section you will find an overview over the materials. Feel free to contact me, if you need anything else or if you have any questions regarding my work!

The Theoretical Paper

The theoretical part of the BA deals with sensory plasticity in general. Available in German only.

The Practical Paper

This is the practical documentary paper, showing the design process in detail. Available in German only.

Picture Package (140MB)

This package contains high resolution pictures of the prototype and the process, which can freely be used with attribution.

Text Package (1 MB)

This package contains text material such as documentary texts and short descriptions in both German and English.

The Code

coming soon!
The pictures from the camera have to be processed for the output on the glove by software code. This is currently executed on an Apple Macbook in the backpack on the back of the user which runs a C++ script. This specific code and other materials are available under this link on Github.

Contact

I am BA in Integrated Design (Köln International School of Design) and currently working as a freelancer.

Please feel free to get in touch with me: mail@jakobkilian.de or visit my portfolio on jakobkilian.de

Der Autor und Macher des Projekts Jakob Kilian im Porträt
Jakob Kilian
Interaction Designer
Eine Hand steckt Kabel zur Verbindung eines elektrischen Geräts