We think that we see with our eyes, but is that a fact?
|Example spectrogram of a one-second |
sound generated by The vOICe.
Image source: www.seeingwithsound.com
Breaking stereotypes gives many possibilities.
Dr. Meijer about his invention.
Transforming of visual image into sound: how does it work?
However, it's not a magic bullet.
What about the feedback from sightless users?
|On the photo: Pranav Lal and photographs made by him. The source: http://techesoterica.com/|
What are your feelings while perceiving the world via The vOICe? What are the advantages for you personally in using The vOICe?
Pranav Lal: As regards my feelings, I cannot describe them in one word. I experience so much more. For example, I was looking at the staircase outside my house. I have seen the architecture plans of the house using The vOICe. I looked at the staircase sideways with The vOICe and connected the architect’s drawing with what I was seeing. When I was being driven to a shop that was quite far from my house, I was looking at all the vehicles and at the walls on the side of the road as well as other things like vehicles stopped at the red lights etc. I got so much more information. Words do not convey visual information. You need to experience it. In addition, The vOICe helps me with orientation. For example, I can walk in a straight line and not collide with colleagues who are standing in random positions in the office. I feel more in tune with my environment and can acquire information almost as fast as a sighted person. Moreover, it gives me more inclusion with the sighted world. I can point to things and ask people what they are and if people get excited about something, I can look at that thing and participate in the conversation. The thing with The vOICe is that you need to practice and start with small things like looking at the door of your bedroom and evaluating how it looks visually.
For how long do you actively use The vOICe? Do you use it during the whole day, or for a short period? Did you experience any side-effects after usage of this program (e.g. headache)?
PL: I have used it for a maximum of 12 hours without any discomfort. I use the program regularly. I wear the setup on a need basis. For example, on a regular day, I may use The vOICe for 5 or 10 minutes to walk around my office but when I go on holiday or to a new place, I only take it off when I return to my hotel room. I assure you that there are no headaches. There is some discomfort if your setup is not comfortable but we are fixing those problems fast. For example, headphones became uncomfortable for me. I have now switched to bone conduction headphones so my ears are free.
Do you really perceive the soundscapes subconsciously without thinking much about the basic rules of vision-to-sound conversion?
PL: As for subconscious interpretation, I do not consciously think of the rules any more. I sense a scene and then break it down into shapes. I then look at spaces between shapes, patches of light and dark and then look for varying textures. If I encounter something really knew, then I know the 3 basic rules and try to make sense of it. The 3 rules are: the panning represents horizontal placement of an object, the pitch represents the height of an object and the volume represents the brightness of an object.
What will happen is that the more you use the program, the more the rules will become a habit when listening to a soundscape. I frequently find myself using the rules when listening to music and believe me that makes for strange images! I do not exactly build full pictures in my head but more like a functional model a kin to a photographic negative.
Pranav Lal keeps a blog techesoterica.com, where he shares his experience of using The vOICe, as well as his attitude towards other topics.
I would like to note that earlier I wrote about another sensory substitution device – a tactile one named BrainPort. In my opinion, the uniqueness of both The vOICe and BrainPort is that their operating principle is based on our organism’s (brain’s in this case) natural ability to adapt towards new conditions. The sensory substitution devices are noninvasive, relatively cheap and can open up new opportunities in perception of the world that we have not thought of before.
Another point concerning The vOICe that amazed me much (apart from everything else) is that it may give the experience of visual perception to congenitally blind individuals. Thus the specialists know that the concept of ‘critical periods’ exists, which assumes that if during a particular developmental period (that happens in childhood) the visual stimuli do not come to the brain, visual functions do not develop (reviewed in ). This is confirmed by psychological observations of children with vision loss at different ages . For instance, in case the visual deprivation starts at 6 months of age, it prevents the development of normal acuity. If the visual deprivation happens near birth, it prevents sensitivity to the global direction of motion. Nevertheless, the studies of congenitally blind subjects that used The vOICe  as well as Pranav Lal’s experience demonstrate that they still may acquire such visual functions as acuity, shape recognition, object localization in space, etc., despite having had no visual experience during the developmental periods.
I thank Dr. Peter Meijer and Pranav Lal for their help in creation of this article.
0. Michael Proulx at TEDxBathUniversity: https://www.youtube.com/watch?v=2_EA6hHuUSA
1. Dormal G, Lepore F, Harissi-Dagher M, Albouy G, Bertone A, Rossion B, Collignon O (2014). Tracking the evolution of crossmodal plasticity and visual functions before and after sight-restoration. Journal of Neurophysiology, 113, 1727-1742. doi: 10.1152/jn.00420.2014.
2. Collignon O, Dormal G, Albouy G, Vandewalle G, Voss P, Phillips C, Lepore F. (2013). Impact of blindness onset on the functional organization and the connectivity of the occipital cortex. Brain, 136 (Pt 9): 2769-83. doi: 10.1093/brain/awt176.
3. Meijer PB (1992). An experimental system for auditory image representations. IEEE Trans Biomed
5. Striem-Amit E., Guendelman M., Amedi A. (2012). ‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution. PLoS ONE 7(3): e33136. doi:10.1371/journal.pone.0033136
6. Merabet L, Poggel D, Stern W, Bhatt E, Hemond C, Maguire S, Meijer P and Pascual-Leone A (2008). Retinotopic visual cortex mapping using a visual-to-auditory sensory-substitution device. Front. Hum. Neurosci. Conference Abstract: 10th International Conference on Cognitive Neuroscience. doi: 10.3389/conf.neuro.09.2009.01.273
7. Pascual-Leone A,
8. Merabet LB, Maguire D, Warde A, Alterescu K, Stickgold R, Pascual-Leone A.(2004). Visual hallucinations during prolonged blindfolding in sighted subjects. J Neuroophthalmol. 24(2):109-13.
9. Amedi A, Stern W M, Camprodon J A, Bermpohl F, Merabet L, Rotman S, Hemond C, Meijer P & Pascual-Leone A (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience 10, 687 – 689, doi:10.1038/nn1912
10. Manual of The vOICe: http://www.seeingwithsound.com/manual/The_vOICe_Training_Manual.htm
Self-Training for The vOICe: http://www.seeingwithsound.com/training.htm
11. Lewis T. L., Maurer D. (2005). Multiple Sensitive Periods in Human Visual Development: Evidence from Visually Deprived Children. 2005 Wiley Periodicals, Inc., DOI: 10.1002/dev.20055.