American Scientist

A BIOGRAPHY OF THE PIXEL. Alvy Ray Smith. x + 548 pp. MIT Press, 2021. $39.95.

Alvy Ray Smith once declared, in the title of a technical memo, that “A Pixel Is Not a Little Square, a Pixel Is Not a Little Square, a Pixel Is Not a Little Square!” He makes the same argument, with less shouting and table thumping, in his new book, A Biography of the Pixel.

You may well ask, If a pixel is not a little square, then what is it? And those tiny colored squares on the screens we stare at all day—if they are not pixels, what are they?

To answer these questions, Smith takes us on a grand tour, starting in Napoleonic France, where Jean Baptiste Joseph Fourier proclaimed that the world is made of waves. This idea is easy to grasp when it’s applied to a one- dimensional signal such as music, where the sound produced by an entire symphony orchestra can be decomposed into a set of pure sinusoidal waves of various frequencies. Fourier showed that the same process also works in two or three dimensions. For example, patterns of light and dark in a photograph can be represented by waves extending across the width and height of the image.

But waves are not pixels. Sine waves are smooth and continuous; pixels, whether or not they are little squares, are discrete objects. To get from waves to pixels, Smith leaps ahead a century, from the aftermath of the French Revolution to that of the Russian one, when Vladimir Kotelnikov invented the sampling theorem. Kotelnikov showed that you can capture all the information in a waveform without tracing out the details of its undulations. It’s enough to take samples at discrete points, as long as those points are spaced closely enough that the highest frequency component of the wave is sampled twice in every cycle. Those sample sites, Smith informs us, are the true pixels. They are not little squares but dimensionless points where the wave amplitude is measured. From a set of such point-like samples, the complete image (or sound wave, or any other signal) can be fully reconstructed. And the reconstruction is not just a good approximation; it’s exact. No information is lost in the sampling process.

If pixels, properly understood, are sample points, we are left with the question of what to call all those little squares of color that light up the screen of your phone or computer or television. Smith doesn’t have an inspired answer. He suggests the term display element, but for the most part he grudgingly calls them “spread pixels.”

At times, Smith becomes a little testy in his campaign to clarify the meaning of pixel, but the point he is making is important. There’s more at stake than a question of terminology. Pixels have transformed the way we see and think about visual imagery of all kinds, and it’s worth knowing what they are and how they work.

In 1801, when Jacques-Louis David painted Napoleon Crossing the Alps, with Bonaparte astride a white charger, the only way to see the image was to stand before the canvas itself. Smith writes, “a painting and its medium of creation were inseparable.” Pixels changed that: “It became possible to remove a painting, so to speak, from its canvas.” And the essence of that transformation was not slicing the painting into lots of tiny squares of color; the key ideas that liberated pictures from their material media were Fourier analysis and the sampling theorem. Now almost all imagery is digital. Smith comments, “Museums and kindergartens are among the few reliable places to find the analog.”

A Biography of the Pixel is also a biography of scores of people who contributed to the development of these ideas. We follow Fourier as he travels to Egypt in Napoleon’s retinue, and we watch Kotelnikov tiptoe through the ideological minefield of Stalin’s Russia. (He not only survived and avoided the Gulag, he wound up as chairman of the Supreme Soviet.)

Later, when computers enter the story, the pace picks up and the cast of characters becomes crowded with people who adapt the new machines to diverse purposes—entertainment, art, manufacturing, flight simulation, games, publishing, architecture.

Smith’s own contributions have been mainly in the creation of three-dimensional simulations and animations, a process he describes as follows, referring to it as “the Central Dogma of computer graphics”:

A fictitious world is described inside a computer with three-dimensional Euclidean geometry and Newtonian physics. Then it’s observed by a virtual camera that renders its view of the world into two dimensions in Renaissance perspective for display.

The last third of the book is devoted almost entirely to this branch of computer graphics, culminating in the production of Toy Story, the first full-length feature film done entirely with computer animation. The creative work on the film was done at Pixar, a firm that Smith cofounded.

Smith is a diligent historian when it comes to tracking down firsts in computer graphics—the first pixels to appear on a computer screen, the first color pixels, the first 3D animation. Some of these historic firsts are unlikely to be of great interest unless you frequent trivia nights at a really nerdy bar. Others carry greater weight. The chief example is Smith’s decision to introduce the sampling theorem through the work of Kotelnikov. Outside the Russian-speaking world, the theorem is most often attributed to Claude Shannon, who made crucial use of it in his “Mathematical Theory of Communication” article in 1948. Smith makes a strong case for giving Kotelnikov priority; although he wasn’t the first to state the theorem, he was apparently the first to publish a proof, 15 years before Shannon. I had not known that, and I’m grateful to know it now, and also to have learned something of Kotelnikov’s colorful career. (But I still admire Shannon.)

Smith’s pursuit of who-did-it-first becomes somewhat troubling in his treatment of Ivan Sutherland, who led the preeminent U.S. research group in computer graphics at the University of Utah. “Received history,” Smith says, gives Sutherland credit for writing “the first interactive computer graphics program” in 1962 (as part of his work on a PhD thesis supervised by Shannon). Smith’s research revealed that there is prior art for many of the features of Sutherland’s program. The earlier innovators certainly deserve acknowledgment, but being first isn’t everything. Sutherland turned those pioneering ideas into a coherent body of knowledge, which he communicated to a generation of students, including many of Smith’s own close colleagues. Sometimes it’s the last discovery of an idea that counts most, because after that it never needs to be discovered again.

A Biography of the Pixel offers us more than 500 pages overstuffed with history, lore, personalities, technical minutiae, and the adventures of a small band of fanatics obsessed with the dream of making movies out of pure imagination. Another 397 pages of annotations wouldn’t fit in the book, so Smith has made them available for download on his website (alvyray.com/DigitalLight/default.htm). And yet, so much is left out! The book makes no mention of the PostScript page description language, which transformed the technology of publishing in the 1980s and 1990s. There is not a word about JPEG, the format in which most of us keep our pictorial archives, and which is based on a technique akin to 2D Fourier analysis. Also passed over are the various tools of the internet era that have democratized computer graphics; you no longer need technical wizardry and a seven-figure budget to show your pixels to the people.

There’s one more element of the story that Smith might have described in greater detail. When I first picked up A Biography of the Pixel, I guessed that it would also be an autobiography of Alvy Ray Smith. We do learn the outline of Smith’s professional career, from early work on cellular automata to exciting times at the New York Institute of Technology, followed by a stint at Lucasfilm and then the founding of Pixar. But this author who gives such an affecting account of vicissitudes in the lives of Fourier and Kotelnikov is more reticent about his own. Near the end of the story he relates with calm detachment that he was forced to leave Pixar just as production of Toy Story was getting underway. Thus he had to applaud from the sidelines as his lifelong dream of making “The Movie” was completed by friends and colleagues. It’s the final and most poignant reminder of a theme that crops up repeatedly in this narrative—that pixels and all the other abstract tokens of mathematics and technology come to us entangled in very human lives.