American Scientist

One of the most rewarding parts of my work is making connections—first by introducing researchers from disparate fields whose work appears (to me) to have visual commonalities. Other times it is methodologies that share a common thread.

When I visited Michael Cohen, a senior researcher at Microsoft Research, some time ago, he showed me several of their consumer applications. One application being developed as part of the Microsoft Expression designer toolkit under the code name Acrylic includes a "stitching" feature called PhotoMontage, developed to enable amateur photographers to seamlessly and easily stitch together several images (of the Grand Canyon, for example) to create a panoramic view.

Coincidentally, a week before the visit, I had a conversation with John Hart, a graduate student in mechanical engineering at MIT. We discussed the problem of using a scanning electron microscope for samples larger than the instrument's field of view. SEMs, unlike optical microscopes, create images with amazing depth of field, surface contrast and resolution. For those reasons, SEM is the imaging method of choice for many investigators who work with materials having dimension. However, because SEMs are used for the most part to reveal features smaller than the wavelength of visible light, a microscopist who uses an SEM to examine a larger structure—say, 8 millimeters wide, as in the large image at left—cannot possibly get the entire sample into the field of view. Most of the time, the researcher will take a series of images and painstakingly stitch them together by hand in an application such as Adobe PhotoShop. The process is tedious and time-consuming.

Introducing John and Michael seemed obvious, and the results were fruitful. I am convinced that such connections can advance the way we visually document and represent research. I welcome American Scientist readers to get in touch with me if they have their own thoughts about connecting methodologies which initially appear to come from different worlds. Below is my own stitching attempt, this one of pieces of conversations with John and Michael.

F. F. Michael, how does the program work?

M. C. The program finds common features in the images and then aligns them by applying what is called a "homography" to position and stretch each image before blending them. Because the structures are fundamentally three-dimensional, parallax makes it very hard to stitch them. To get the best results John had to take lots of images overlapping by, say, three-fourths of the image (ideally even more). Imagine passing a video camera slowly over the sample. Then you can rely on stitching together only the centers of the images, where there will be less foreshortening.

F. F.  John, if I remember correctly, some of the initial attempts with other software would work only with images taken from one vantage point, not one where you made images from various planar views as you moved the SEM stage.

J. H. Yes, we first tried another application that could only handle a fixed point. Then I tried another, with similar disappointing results and besides, the code was much slower than Acrylic. There was also lots of manual configuration—like picking the common points to align the images—and under most cases it wouldn't even stitch. Finally, as Michael suggested, I took a number of overlapping images as I moved the SEM stage. Acrylic was able to align the images automatically ... you just throw all the frames on the canvas and hit "OK."

F. F. Why did you decide to image the sample with an SEM and not an optical microscope?

J. H. First, the substrate is polished silicon and looks like a mirror, so an optical image is badly confused by reflections of the structures in the substrate. Second, the structures are optically black, so an optical image wouldn't reveal the curves and sharp edges of the structures, as seen in the perspective view. Third, SEM is capable of much higher resolution than an optical microscope by a factor of a few hundred.

F. F. Do you see any problems using this stitching program?

J. H. The stitching program worked very well with these images. I was concerned that the program would have difficulty aligning the edges of neighboring frames, because translating the SEM stage to take each frame slightly changes the perspective. The program slightly warped the frames to fit them together, but this is hardly noticeable in the stitched images.

F. F. How do you feel about my cleaning the final image for this article?

J. H. It looks nice; however, the bits you removed are strands of nanotubes that grow from silicon chips left by cracking the silicon wafer. These appear in the individual frames, and each one consists of hundreds or thousands of nanotubes.