American Scientist

Science strives to be objective, whereas art often comes across as shamelessly subjective. But as any self-aware scientist knows, true objectivity is a goal that is often out of reach. There is a big disconnect between what is ultimately published and translated to the general public under the heading of science and the actual process of experimentation, data collection, and interpretation. An outside observer might be surprised at how messy the work of science actually is, compared with the public perception of the scientific endeavor. In that sense, it’s much more like art production than most people realize.

Subjectivity in science starts with the very choice of question that a scientist sets out to answer. Our biases, especially unconscious ones, then influence our work every step of the way. The scientific method and the importance of reproducibility, converging evidence, and multiple lines of research all contribute to insert objectivity back into the process. But as replication failures mount in virtually every field of science, it’s becoming harder and harder to argue that science and art are at opposite ends of a spectrum.

Within the art world, there is also a blurring of that distinction, as some artists have chosen to insert science as a topic or a process into their work. One innovator of this kind is Deborah Aschheim, a pioneer in the latest trend of cross-pollination between artists and scientists.

For the last decade or so, Deborah’s work has dealt with memory, the loss of memory, and place, three subjects that are linked by the brain region that governs them (and that also happens to be the focus of my own research). Deborah is no stranger to neuroscience: Throughout her career, her pieces have been informed by cutting-edge scientific research on the topics that her art depicts. One of her recent works, a chandelier titled Neural Architecture, portrays the intricate signalling pathways of the human brain by means of a transparent network of tubes, wiring, electronics, and lights, housed in forms sewn out of clear plastic bath mats. Another installation, Eavesdrop Network, uses baby monitors to “listen” to other artists’ creations in the same gallery, linking the pieces together in a work of sound art. In 2009 she was named as the inaugural Hellman Visiting Artist at the Memory and Aging Center (MAC) at the University of California, San Francisco.

I met Deborah when I was a postdoctoral fellow at the MAC, where I was investigating the paradoxical emergence of creative impulses in some of our patients. These were people who, upon losing the ability to communicate using language, begin to find other ways of expressing themselves. Many were suffering from a neurodegenerative disease called semantic dementia, which slowly robs a person of conceptual knowledge; these patients tended to spend more and more time engaged in activities that are highly visual as their disease progressed.

Although called semantic dementia, this is a loss not simply of words but of concepts. At first, patients might have trouble finding the right word to use as they try to communicate, but as the disease progresses, their deficits encompass other aspects of semantic memory. These patients show us the key role that conceptual knowledge plays in our ability to understand one another.

In children, the knowledge of conceptual categories develops hierarchically. An 18-month-old toddler might learn the name of one exemplar in a category: A furry, four-legged creature that barks is a dog. The child might then confuse dogs with other four-legged creatures like cats, goats, and horses until, at about 24 months, he learns that cats and dogs are exemplars of a higher-order category of animals or even pets. With expertise, we can eventually distinguish between many different dog breeds.

In patients with semantic dementia, the loss of conceptual knowledge follows the reverse pattern: First, a patient might not be able to tell the difference between a beagle and a Labrador. Later, cats and dogs might be confused. Eventually, distinguishing between animals becomes impossible. The speech of these individuals is littered with general nouns and nonspecific words: this, that, there, stuff. At the same time, in a subset of patients, a drive to produce unique visual art dominates their daily activities.

I wondered whether we might gain some insight into these patients’ experience by seeing the world from their perspective. To that end, I devised an observational experiment—admittedly, a bit of a fishing expedition—to give us some idea of what they might be paying attention to as they interact with their environment. I asked them to look at various images, ranging from landscapes to detailed images of simple objects and even iconic works of art, while I used an infrared camera to track their eye movements and fixations.

What became clear straight away was that the semantic dementia patients paid attention to parts of the images that the other groups largely ignored. Whereas most other patients and healthy controls looked at salient parts of the pictures—the vanishing point, for example, or an animate object like a penguin or a person—patients with semantic dementia each seemed to have a unique pattern of fixations, largely ignoring the conceptual framework of an image. For example, when shown a picture of train tracks receding into the distance, most of us spend a disproportionate amount of time gazing at the vanishing point, seemingly waiting for the train. But the patients with semantic dementia treated that part of the image as no more interesting than any other part.

Neurologists can use eye movements as part of their diagnostic criteria, because the different diseases have different effects on the parts of the brain that control eye movements. But semantic dementia patients do not stand out on the neurological exam as Alzheimer’s disease patients do, for example. Instead, patients with semantic dementia retain healthy control of their eyes—it is what pulls their attention in the visual field that changes.

As it happens, Deborah had an interest in aphasia, the categorical term that subsumes language disorders; her aunt had suffered from aphasia. When I told Deborah about our observational experiments, she wanted to see the results right away. And I was excited to show a visual artist aspects of my work that I found to be aesthetically interesting but that would never be observed by the general public.

When I showed Deborah the videos of the different eye movements carried out by patients and healthy subjects, she was most struck by the sheer volume of data. In any given experiment, most scientists collect far more data than is ever published or even analyzed fully. As we navigated through the maze of folders on my computer, she began to wonder what happens to all those files. In fact, the individual eye tracking movies, the tables of statistical analyses, the matrices of fixation points, and so many other files unique to a study are simply stored away until, years later, the drive fills up and storage becomes a problem.

To Deborah, though, these data were the raw materials of an invitation to creativity. She gives these data a second life by incorporating some of them into art pieces, in forms and patterns that require no specialized scientific knowledge to appreciate.

The waiting room of our neurology clinic displays one such piece, called Eye-Tracking. Two video monitors are embedded in a large depiction of a neural network, which is draped across a wall. One monitor plays videos of my lab’s eye-tracking data: Eye movements of patients and healthy controls are displayed as a moving red line superimposed over the picture they are observing. The other monitor shows two eyes moving in tandem with the results of the eye-tracking measurements.

To a scientist, data are precious. Often data analysis has the appeal of a treasure hunt, complete with the rewards that come with new discoveries. That’s partly why it’s so hard to let it go. And data visualization can be its own artistic endeavor, with many of us spending hours writing code to help us see the forest for the trees.

Deborah is interested in transforming the stressful, boring experience of the waiting room into a time when patients might be educated and connect with the larger mission of the institution. “The eye tracking was so appealing because it told a story anyone could understand, since the data were so visual,” she explains. The artwork, she hopes, will open up the process of research: giving the patients access to data they don’t usually see, elevating them to the status of participants rather than subjects, and helping them understand that their medical options are still evolving.

What Stands Out

In another collaborative project, we devised a work of art to show just how beautiful and personal data collection can be. Using functional magnetic resonance imaging (fMRI), I tracked neural activity while Deborah and fellow artist Lisa Mezzacappa interacted with their own art and with carefully curated lures. During the data analysis, I grew frustrated by the constraints of neuroimaging: Deborah and Lisa have skull sizes that are very different, and it was difficult to create overlaying statistical maps that showed activation in comparable brain regions.

At one point I brought Deborah into the lab, flipped through brain scans, and tweaked the statistical threshold for activation to give her a sense of what the raw data actually look like. She remarked how different one result looked from another, each one created by changing just one number in the statistical tool. We talked through different strategies for choosing a statistical threshold. Although “massaging the data,” as scientists call this procedure, should be strictly avoided to preserve objectivity, it is a more common practice than many will admit.

With the aim of demonstrating this subjectivity in a work of public art, we created a piece called Thresholds of Significance. The installation is a series of videos, comparing statistical maps of brain activation to the emergence of lights in a city skyline at dusk. As the sun begins to set, some features of the skyline capture the eye while others are ignored. Then, as the sky darkens and the lights turn on, the cityscape takes on a different form. Some of the salient parts brighten—these can be thought of as the brain areas important to the task at hand, no matter what the threshold might be—while other parts fade into oblivion. Still others come to the fore only when the signal-to-noise ratio is at its highest, when the sky is completely dark and the only light available is artificial. These videos alternate and are accompanied by a soundtrack composed by Lisa, including some of my own vocals.

We have presented this installation in a handful of venues, and the reactions from the audience have been revelatory. Most people take the objectivity of neuroimaging for granted and are surprised to see just how changeable the brain maps are. Like our art installation, the results of the eye-tracking observational study remain unpublished. They have, however, served as the impetus for a follow-up study that was eventually published in the journal Neuropsychologia.

Because our patients seemed to ignore the salient aspects of the images, we wondered whether there might be a visual task at which they could excel but which healthy individuals found difficult. A real-world example of such a task is the Where’s Waldo series by Martin Hanford, in which the goal is to find the character Waldo, embedded in a complex visual scene. Because Waldo shares features with many other distracting foils, a healthy person needs to search each part of the image, serially, until he finds the intrepid traveler.

But what if the distracting items could be ignored? In that case, no matter how many items were in the array, performance should remain constant. And that’s exactly what we found with our semantic dementia patients: In a version of the Where’s Waldo search task, patients outperformed their healthy counterparts by being just as accurate and quick to find the target when the array contained 30 distracters as when it contained 5.

By looking for pathological changes in the brain that might lead to a burst of visual creativity, we found a task at which patients with semantic dementia excelled. Perhaps even more importantly, the unpublished data that led to this discovery now adorn waiting rooms and other spaces, helping patients and their caregivers connect with the larger goal of neuroscience: to understand the brain basis of the human experience.