by Ken O'Sullivan
In 1979 Pat Fletcher lost her sight in a horrific industrial accident. Today she has emerged as a trailblazer in the use of an amazing technology, which, in a very real sense, has allowed her to see again by using her ears. Not long ago I had the chance to talk with Pat, who lives in Buffalo, N.Y.
Her journey started unexpectedly in the late '90s, and had more to do with her wardrobe than anything else. "I wanted to discover a means where I could use my scanner to detect color ... So I went on the Internet and found the Voice. At that time I saw that it was a color detector; I wasn't interested in any of the other stuff, nor did I even understand the other capabilities of the program. So I downloaded it, put my shirt on the scanner, and, voila!"
Developed by a Dutch software engineer, Dr. Peter Meijer, the "other stuff" Pat mentions is a powerful method of converting visual images into sound. Soon Pat was exploring sample soundscapes from Meijer's web site. Although the results were intriguing enough to keep her going, she had yet to find the key to unlock her understanding, until she took matters into her own hands. She scanned a plastic cup, and used that picture with the voice program. "I turned the program on where I could use the arrow keys. Then while arrowing up and down on different spots on the soundscape I ran my finger along the cup. I applied what I was hearing to what I was touching, and then it made sense."
Pat's principle breakthrough came when she determined to take the Voice beyond the four walls of her study. She searched the Internet for a small video camera that she could use with a notebook computer. "When I did that, that unchained me. I took a tiny webcam and put it on the end of a ball cap, and put the notebook in a backpack, and put on a headset. I really wasn't expecting anything. I thought 'who knows what's going to happen here.' I walked out of my room, and looked down the hall, and nearly went down to my knees because I could actually see the blinds that were hanging on the side of the wall. Now, mind you, I did know there were blinds hanging in the hall, but you forget about every little aspect of your environment when you don't see it. It just brought tears to my eyes, and from that point I decided I'm going to use this stuff. I'm going to work with it and wear it. That's when I discovered that the more you work with the program, the more your body could understand and relate to what you were seeing."
To be clear, the Voice software does not produce a "voice" at all. It is neither human nor computerized speech. Those close to it say learning to use the Voice is akin to learning a new language: The meaning is there, but it doesn't jump right out at you.
While the technology may be complex, the concept is fairly simple. Here's how it works: A small video camera is used to capture an image. The camera sends that visual information to a computer. The Voice software then converts the picture into a dynamic, multi-layered stereo sound with a rather science-fiction-like quality. Pitch is used to indicate physical elevation: a higher pitch for up and a lower pitch for down. Volume is used to represent brightness: the brighter the image, the higher the relative volume. Thus, when perceiving one's front yard through earphones, a tall oak tree on the right will create a distinctly different impression than a sunlit driveway on the left. At present, the computer-generated soundscape refreshes about once every second. Of course no description here can substitute for actually hearing these sounds (more on that at the end of this article).
Pat explains, "You can develop a sense of the soundscapes within two to three weeks. Within three months or so you should start seeing the flashes of your environment where you'd be able to identify things just by looking at them."
In truth, this is the essence of the seeing with sound concept. When Pat speaks of "seeing," that's exactly what she means. She's emphatic: "It is sight," she says. "I know what sight looks like. I remember it."
As controversial as this assertion may be, as incredible as it may sound, this claim isn't merely subjective or anecdotal; it's real. And it is only one facet of a growing body of knowledge known as neuroplasticity, which shows that the brain is far more malleable, far more adaptable than science has long assumed.
About seven years ago, Pat (along with another individual, blind from birth) took part in a Harvard study. By using Magnetic Resonance Imaging, a leading researcher in the field, Dr. Alvaro Pascual-Leone, found that when sound is used in this way it stimulates not the auditory centers of the brain but the areas of the brain that convert sensory information into vision.
Pat says, "They actually took pictures of our brains as we listened to different soundscapes, or touched an object, or heard a sound like a whistle. What it proved, after they went through all that imaging, is that my brain lit up in the same way that a sighted person's brain lights up when they see."
To illustrate the power of the Voice system in her day-to-day life, Pat described a recent car trip. She was able to see passing cars, lines on the road, even differences among those lines. From her place in the back seat, she could perceive the driver's hands on the steering wheel, and the appearance of the dashboard.
She explains, "After a point the soundscapes are no longer soundscapes. You don't see or hear soundscapes; you see vision ... actual vision. I am never not amazed when I put on the gear. At any time I could have an experience of sight. Of course I have continuous sight in areas that I know, but there are times when I'm looking at stuff and I don't know what it is."
Pat likes the fact that the Voice system is easy to set aside. Incredibly, this is something she refers to as "going blind." "If I don't want to see, if I just want to watch TV, I can take it off. I can lay it down. Now, it takes me a minute to become blind again. I can't just take it off and ... boom. It takes my brain a few minutes."
A gadgeteer by nature, Pat has spent years mixing and matching components to make her gear ever more inconspicuous and comfortable to wear. She's an expert. Her contacts with the scientific community and the media have established her as a kind of ambassador for the seeing with sound concept. She is intelligent, articulate, and enthusiastic. Her optimism is palpable, and with good reason.
Seeing with sound now seems to be on the threshold of a substantial new development, one that could move it into the mainstream. A pair of computer scientists, working in Pasadena, Calif., have formed a company called MetaModal, LLC. Using the Voice as a starting point, their ultimate goal is to integrate software and hardware into a single, commercially viable unit, a sophisticated pair of sunglasses, and to train individuals in its use. Working under a grant from the National Science Foundation, Dr. Enrico Di Bernardo and Dr. Luis Gonçalves put together a prototype. They've tapped Pat's knowledge and experience, and are working with orientation and mobility specialists to train Voice users and learn what works best.
Di Bernardo says, "We really want to make a difference to the end user, to empower them for their independent living."
Di Bernardo, who speaks in a vibrant Italian accent, says a device of the kind they're envisioning is only made possible by technological advances over the last 10 years, especially in regard to cell phones. "We are piggybacking on every single part of it, from the cameras to the batteries to the electronics. If cell phones didn't push the boundaries of miniaturization, we wouldn't be able to do this."
MetaModal is a term the Pasadena team borrowed from Dr. Pascual-Leone. "He coined this term," Di Bernardo says. "The idea being that the way we perceive the world doesn't happen in any specific mode, or sense, but it really happens at the junction of all senses."
I thank Pat Fletcher and her friend Carol Bartlett for exposing me to this marvelous topic. I thoroughly enjoyed my conversation with Pat and Carol, who were very generous with their time on a Sunday afternoon.
Before our talk ended, I asked Pat to indulge me in a silly exercise. Without saying why, I asked her to complete this sentence: "Using the Voice system is ..." Her response was firm and immediate: "the pathway to sight."
For Pat Fletcher, that seems to say it all.
Those interested in learning more can visit www.seeingwithsound.com. This site is the place to go to download the Voice software at no charge. It's also a rich source of information. Following the "news" link, for example, leads to a well-maintained collection of annotated links to press coverage and events. Also, the Canadian Broadcasting Company produced a delightful 22-minute radio segment on the Voice system. It features Pat and what she's hearing. It also includes more about Dr. Alvaro Pascual-Leone, and the Harvard brain study. To hear it for yourself, visit www.cbc.ca/quirks/archives/04-05/apr02.html. And www.metamodal.com is the site of the Pasadena startup, which includes a number of videos of the system in use.