Sunday 13 September 2015

Seeing with sound: what hidden abilities do we have?

Author: Olena Markaryan                                     


We think that we see with our eyes, but is that a fact?


The experimental psychologist Dr. Michael Proulx at the TEDx Talks once said: “We think that we see with our eyes, but in fact we see with our brain. Our eyes just provide us the information and the brain sorts out that information, makes sense of it and makes that feeling of seeing” [0]. These words give us a lot of food for thought as well as push us to revaluate our abilities. What happens if the visual input is cut off? Can the person still restore visual perception in this case?
Dr. Ione Fine, talking about the crossmodal plasticity phenomenon in her interview, explains that if a person doesn’t get visual input, the part of the brain that is responsible for the processing of visual input can’t just stop functioning and do nothing. Instead, it starts to be fed with auditory and tactile information for analysis. Actually, in the daily life of sightless people the visual cortex is actively engaged in the audio information processing [0]. This was confirmed by Dr. Giulia Dormal and Dr. Olivier Collignon et al.’s observations when applying functional magnetic resonance imaging (fMRI). The scientists noted the activation of visual cortex regions in response to audio stimuli in congenitally blind and late-onset individuals [1, 2].
Example spectrogram of a one-second
sound generated by The vOICe.

Image source: www.seeingwithsound.com
 

 

Breaking stereotypes gives many possibilities.


Thus, if one sensory receptor is not available, why not to use the other one in order to provide the brain with necessary ‘food’ for processing? Actually, this principle has been used by Dr. Peter Meijer who developed a system converting visual images to auditory signals in 1992 [3].  This system provides the user with ‘visual’ information via the sense of hearing and is called The vOICe [4]. With help of this sensory substitution device and after extensive training, totally blind individuals are able to differentiate between the shapes of different objects, identify actual objects, and also locate them in space, identify and mimic the body posture of a person standing a few meters away, navigate in crowded corridors while avoiding obstacles, and even deduce live, 3-dimensional emotional facial expressions from the shape of the face and mouth [5].
Another interesting fact was found by Dr. Ella Striem-Amit et al. The researchers decided to evaluate the visual acuity which The vOICe provides to 8 congenitally and 1 early-onset totally sightless individuals. Importantly, the subjects were trained to use the program during several months (2 hours a week) prior the acuity assessment. With help of the Snellen tumbling-E test it was revealed that the visual acuity of participants varied between 20/200 and 20/600. Moreover, 5 of 9 participants had a visual acuity exceeding the blindness threshold as established by the World Health Organisation at 20/400. Therefore they could now formally be regarded as low-vision sighted [5].

Dr. Meijer about his invention.

Being intrigued with The vOICe operating principles, I couldn’t resist asking the device developer some questions:

Dr. Meijer, taking into account that seeing with sound is a quite unordinary approach to perceive the external world, how did the idea that people can actually see with sound come to your mind?

Dr. Peter Meijer: In the brain, at the neuron level, the signals carrying visual or auditory information all look the same (just spike trains), so if the switch-box circuitry permits there could be a "leaking" of auditory input to the visual brain areas. [In similar way as] With the old telephone system with copper wires in the ground, you can call people that you have never called before, all without changing the physical wiring in the ground.

How and when does sound start to be analyzed by visual cortex instead of (or after) auditory cortex?

Dr. P.M.: Within days of complete blindfolding of normally sighted people, the visual cortex starts to respond to sound, cf. 6, 7, 8. How it works, and the extent to which for instance the parietal cortex (association cortex) is involved is still unclear. The basic idea is that the visual cortex "likes" to do what it is good at, such as doing spatial computations, and if it can (and must, for lack of eyesight) get that information elsewhere, the brain will adapt. For similar reasons, in Charles Bonnet syndrome, loss of eyesight leads to visual hallucinations because the visual areas in the brain still "want" to create realistic visual renderings. Ideally, sensory substitution would replace meaningless hallucinations by visual hallucinations based on true visual input, even though that input is now differently encoded (e.g. in sound). Normal vision can be viewed as visual hallucinations where the content just happens to match physical reality because the content is derived from environmental visual input from the eyes.

Transforming of visual image into sound: how does it work?


Visual images are captured by a camera and then transformed into so-called soundscapes that preserve the object’s shape information. The algorithm of visual-auditory conversion is the following: time and stereo panning form the horizontal axis in the sound representation of an image, tone frequency makes up the vertical axis and loudness corresponds to pixel brightness [9]. For example, a bright dot gives a short beep, with pitch telling elevation; a rising bright line gives a rising tone. More examples and corresponding soundscapes you can find in the manual to the program [10].
Remarkably, The vOICe lets one use the natural optical features, namely visual perspective, parallax, occlusion, shading, and shadows which may help greatly in the independent navigation. For example, knowing the rule that an object appears twice as large at half the distance and applying it while moving around and analyzing soundscapes, the user can differentiate between and identify nearby obstacles as well as distant landmarks. More information about the interesting regularities you can find in the manual for the The vOICe [10]. 

On the photo: Setup for the Windows version of The vOICe. The vOICe for Android application is also available. Source of images: www.seeingwithsound.com

To start practicing The vOICe you only need a computer to install the free-to-download Windows program, and headphones. This will let you practice the interpretation of soundscapes of simple shapes. When you are ready to go to the next level, you will need to use a portable computer (laptop or tablet) and a camera to get a live view of the visual environment. All the details about software and hardware you can find at seeingwithsound.com. Also, you can find recommendations there regarding the use of bone conduction headphones (which permit hearing both the soundscapes and natural environmental sounds) and USB camera glasses which will make practicing with The vOICe more convenient.

However, it's not a magic bullet.


It is highly important to undergo the step-by-step training before using The vOICe in a real environment, especially outside. Listening to The vOICe soundscapes of the outer environment without any preliminary training may cause irritability or a headache in some cases because of the stream of complex sounds which you cannot interpret yet. A quite apt comparison that Dr. Meijer once made (personal correspondence): “Learning to drive a car can initially be highly stressful too, with the need to near-simultaneously watch the road, watch the rear view mirror, and operate the gas, gear lever, clutch and steering wheel in real-time. Still, would-be drivers are not complaining [and keep on studying]. Mastering The vOICe means hard work and persistence”. Here you can find suggestions for  self-training, and the English manual (a translation into Russian is also available) will help you to further explore usage of this program.
Dr. Meijer’s advice regarding being persistent with The vOICe training is corroborated with scientific observations. Dr. Lotfi Merabet et al. measured brain activity (using fMRI) before and after The vOICe training. Before the training, 4 sighted subjects showed strong activation of auditory cortex but no activation of visual areas in response to The vOICe audio stimuli. After one week of training, activation was also recorded in visual cortical areas in 3 out of 4 of the sighted subjects [6]. Other interesting results include what Dr. Amir Amedi et al. observed while studying the lateral-occipital tactile-visual area (LOtv), which is normally responsible for object shape recognition via integrated visual and tactile information processing. According to fMRI results, the soundscapes generated by The vOICe also activated LOtv during shape recognition, whereas other sounds still did not activate LOtv. Moreover, the LOtv activation was only observed in subjects who were trained to interpret the soundscapes. The scientists added that it is unlikely that visual imagery instigates the processing of information from soundscapes in LOtv [9].


What about the feedback from sightless users?

On the photo: Pranav Lal and photographs made by him. The source: http://techesoterica.com/
I decided to contact a sightless person who uses The vOICe in his daily life. Recently, the New Scientist published an article about the congenitally blind young man Pranav Lal who makes wonderful photos of places he travels to. He uses The vOICe to make good shots. Mr. Lal has been using The vOICe since 2001 (i.e. 14  years as of now).  So I supposed he was the right person to ask for opinions regarding the sensory substitution device:

What are your feelings while perceiving the world via The vOICe? What are the advantages for you personally in using The vOICe?

Pranav Lal: As regards my feelings, I cannot describe them in one word. I experience so much more. For example, I was looking at the staircase outside my house. I have seen the architecture plans of the house using The vOICe. I looked at the staircase sideways with The vOICe and connected the architect’s drawing with what I was seeing. When I was being driven to a shop that was quite far from my house, I was looking at all the vehicles and at the walls on the side of the road as well as other things like vehicles stopped at the red lights etc. I got so much more information. Words do not convey visual information. You need to experience it. In addition, The vOICe helps me with orientation. For example, I can walk in a straight line and not collide with colleagues who are standing in random positions in the office. I feel more in tune with my environment and can acquire information almost as fast as a sighted person. Moreover, it gives me more inclusion with the sighted world. I can point to things and ask people what they are and if people get excited about something, I can look at that thing and participate in the conversation. The thing with The vOICe is that you need to practice and start with small things like looking at the door of your bedroom and evaluating how it looks visually.

For how long do you actively use The vOICe? Do you use it during the whole day, or for a short period? Did you experience any side-effects after usage of this program (e.g. headache)?

PL: I have used it for a maximum of 12 hours without any discomfort. I use the program regularly. I wear the setup on a need basis. For example, on a regular day, I may use The vOICe for 5 or 10 minutes to walk around my office but when I go on holiday or to a new place, I only take it off when I return to my hotel room. I assure you that there are no headaches. There is some discomfort if your setup is not comfortable but we are fixing those problems fast. For example, headphones became uncomfortable for me. I have now switched to bone conduction headphones so my ears are free.

Do you really perceive the soundscapes subconsciously without thinking much about the basic rules of vision-to-sound conversion?

PL: As for subconscious interpretation, I do not consciously think of the rules any more. I sense a scene and then break it down into shapes. I then look at spaces between shapes, patches of light and dark and then look for varying textures. If I encounter something really knew, then I know the 3 basic rules and try to make sense of it. The 3 rules are: the panning represents horizontal placement of an object, the pitch represents the height of an object and the volume represents the brightness of an object.
 What will happen is that the more you use the program, the more the rules will become a habit when listening to a soundscape. I frequently find myself using the rules when listening to music and believe me that makes for strange images! I do not exactly build full pictures in my head but more like a functional model a kin to a photographic negative.
Pranav Lal keeps a blog techesoterica.com, where he shares his experience of using The vOICe, as well as his attitude towards other topics.

Afterword.


I would like to note that earlier I wrote about another sensory substitution device – a tactile one named BrainPort. In my opinion, the uniqueness of both The vOICe and BrainPort is that their operating principle is based on our organism’s (brain’s in this case) natural ability to adapt towards new conditions. The sensory substitution devices are noninvasive, relatively cheap and can open up new opportunities in perception of the world that we have not thought of before.
Another point concerning The vOICe that amazed me much (apart from everything else) is that it may give the experience of visual perception to congenitally blind individuals. Thus the specialists know that the concept of ‘critical periods’ exists, which assumes that if during a particular developmental period (that happens in childhood) the visual stimuli do not come to the brain, visual functions do not develop (reviewed in [11]). This is confirmed by psychological observations of children with vision loss at different ages [11]. For instance, in case the visual deprivation starts at 6 months of age, it prevents the development of normal acuity. If the visual deprivation happens near birth, it prevents sensitivity to the global direction of motion. Nevertheless, the studies of congenitally blind subjects that used The vOICe [5] as well as Pranav Lal’s experience demonstrate that they still may acquire such visual functions as acuity, shape recognition, object localization in space, etc., despite having had no visual experience during the developmental periods.

Acknowledgement.

I thank Dr. Peter Meijer and Pranav Lal for their help in creation of this article.
 

References:
0. Michael Proulx at TEDxBathUniversity: https://www.youtube.com/watch?v=2_EA6hHuUSA
1. Dormal G, Lepore F, Harissi-Dagher M, Albouy G, Bertone A, Rossion B, Collignon O (2014). Tracking the evolution of crossmodal plasticity and visual functions before and after sight-restoration. Journal of Neurophysiology, 113, 1727-1742. doi: 10.1152/jn.00420.2014.
2. Collignon O, Dormal G, Albouy G, Vandewalle G, Voss P, Phillips C, Lepore F. (2013). Impact of blindness onset on the functional organization and the connectivity of the occipital cortex. Brain, 136 (Pt 9): 2769-83. doi: 10.1093/brain/awt176.
3. Meijer PB (1992). An experimental system for auditory image representations. IEEE Trans Biomed Eng. 39(2):112-21.
5. Striem-Amit E., Guendelman M., Amedi  A. (2012). ‘Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution. PLoS ONE 7(3): e33136. doi:10.1371/journal.pone.0033136
6. Merabet L, Poggel D, Stern W, Bhatt E, Hemond C, Maguire S, Meijer P and Pascual-Leone A (2008). Retinotopic visual cortex mapping using a visual-to-auditory sensory-substitution device. Front. Hum. Neurosci. Conference Abstract: 10th International Conference on Cognitive Neuroscience. doi: 10.3389/conf.neuro.09.2009.01.273
7. Pascual-Leone A, Hamilton R (2001) The metamodal organization of the brain. Prog Brain Res 134: 1–19.
8. Merabet LB, Maguire D, Warde A, Alterescu K, Stickgold R, Pascual-Leone A.(2004). Visual hallucinations during prolonged blindfolding in sighted subjects. J Neuroophthalmol. 24(2):109-13.
9. Amedi A, Stern W M, Camprodon J A, Bermpohl F, Merabet L, Rotman S, Hemond C, Meijer P & Pascual-Leone A (2007). Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nature Neuroscience 10, 687 – 689, doi:10.1038/nn1912
Self-Training for The vOICe: http://www.seeingwithsound.com/training.htm
11. Lewis T. L., Maurer D. (2005). Multiple Sensitive Periods in Human Visual Development: Evidence from Visually Deprived Children.  2005 Wiley Periodicals, Inc., DOI: 10.1002/dev.20055.

No comments:

Post a Comment