American Scientist

A peculiar detail in the famous painting The Night Watch shows that Rembrandt van Rijn was not only a great painter but also an astute observer of human behavior. 

Off to the left of center, half in shadow, stands a red-uniformed guard who holds the ramrod for his musket in an uncomfortable-looking, thumb-down position. Why did Rembrandt pose the guard this way? A thumb-up posture would have probably allowed the guard to have greater power and more control. But there is another consideration: a thumb-down posture would put the guard in a stronger or more resolute-looking pose when he withdrew the rod from the barrel.

We suspect Rembrandt wanted to convey that the guard, along with other members of the night watch, would be in a strong position to defend the Netherlands. The artist may have understood that the way one grasps an object depends not just on the properties of the object at the time of the grasp, but also on what one plans to do with the object. Rembrandt may have sensed that what’s in a grasp is in large part what’s in the mind of the grasper.

As cognitive psychologists, we are interested in the mind. We analyze behavior to gain insight into the intelligence behind it. In particular, we study how people and animals grasp objects. We do this to learn about action planning and control, and we pursue this topic not just for theoretical reasons but also for practical ones. Roboticists may find our work useful for informing the design of more adept artificial agents. Healthcare professionals may find our work relevant for informing medical diagnosis and rehabilitation. Regulators, managers, and engineers may benefit from our work insofar as it may inform the design of skill-training systems and safer and more efficient setups for work, transport, and play.

Grasp planning is exquisitely sensitive to immediate and forthcoming demands.

• Facebook
• Twitter

We believe that a useful way to study action planning is to observe changes in behavior as a function of the behavior that follows. If an action differs depending on the subsequent action, the anticipatory effect can be said to reflect planning. Anticipatory changes of this sort have been studied in depth in speech production—for example, in rounding the lips before pronouncing the “t” in “tulip” (as opposed to the “t“ in “tailpipe,“ for example)—a phenomenon called coarticulation. We have been among the first to focus on the manual analog of such changes, which we call comanipulation.

Two naturalistic observations form the basis of our research. One concerns the orientation of the hand as it reaches out to grasp something; the other concerns the height of the grasp. The first observation came from a common scene: a waiter in a restaurant filling glasses with water. To begin with, the glasses were upside down on a table. The waiter took hold of each glass not with his thumb up, which is the usual way to grasp a glass, but with his thumb down. (Throughout this article, we use the terms “thumb-up” or “thumb-down” to refer to the base of the thumb.) After grasping each glass, the waiter turned it upright, held it with his thumb up while pouring, and then set the filled glass back down on the table, as demonstrated by our student volunteer in the photographs at right. Presumably, the waiter used a thumb-down grasp to allow him to complete the task using postures that were comfortable and easy to control.

According to this interpretation, the waiter’s grasp was shaped not only by the visual appearance of the glass (an instance of what we call first-order planning), but also by what he planned to do with the glass subsequently (an instance of what we call second-order planning). Like the guard in The Night Watch, the waiter apparently changed his grasp according to what he planned to do next.

Insights from the Lab

We have conducted several laboratory experiments to test this hypothesis. (Here and throughout this article, when we summarize research without citing particular authors, the work is described in a 2012 Psychological Bulletin review article cited in the Bibliography.) In one experiment, illustrated at right, university students reached out to turn a handle 180 degrees from each of eight cardinal directions. All the participants were right-handed, but half were instructed to use their right hand, while the other half were instructed to use their left hand. When we looked at the range of positions tested, we found that for the right-hand group, the probability, p(T), of grasping the handle with the thumb toward the pointer was lowest when the final pointer position was near position 4, whereas for the left-hand group, p(T) was lowest when the final pointer position was near position 6. Participants grasped the handle with the thumb in a way that helped them avoid extreme arm positions at the ends of the rotations. To accomplish this, they sometimes adopted extreme arm positions at the starts of the rotations.

Further experiments indicated that ease or comfort underlies this planning effect. In one study, university students rated how comfortable their hand felt while holding the handle at each cardinal angle. The positions where participants were least likely to align their thumbs with the pointer at the ends of the rotations were also the positions that got the lowest comfort ratings.

In another experiment, students turned a handle that had a latch attached to it so that the handle locked when it reached the target. This arrangement contrasted with the one in the previous experiments, where the handle needed to be steadied at its final position because the wheel on which the handle was mounted had very little friction. When participants did not have to carefully control the final position of the wheel, they put much less emphasis on ending comfortably, a result that suggests the choice of grasp reflected a desire for control at the time of task completion.

Consistent with this interpretation, other experiments showed that participants could move their extended forearms much more quickly at midrange arm angles than at extreme arm angles and that the variability was much lower midrange than at the extremes. These results help explain why participants preferred midrange arm angles over extreme arm angles when they had to do careful aiming.

The second observation, which concerns the height at which an object is grasped, was made in the homeliest of circumstances. One of us (Rosenbaum) needed to use a toilet plunger to attack a home plumbing problem. Later, he noted that when he took hold of the plunger to return it to the floor where it normally stood, he grasped the shaft very high. Was it possible, he wondered, that the height at which he grasped the plunger facilitated its subsequent control? Do people generally grasp objects high to place them at low positions and low to place them at high positions? They would if they avoided extreme postures when lowering or raising objects.

Bringing the task to the laboratory confirmed that there is indeed an inverse relation between grasp heights and subsequent object-placement heights. The drawings at right illustrate this point in an experiment in which university students reached out to move a plunger from a fixed height to various target heights—some higher and some lower than the starting height. When participants were asked to move the plunger to a lower level, they grasped it high on the shaft, but when they were asked to bring the plunger to a higher level, they grasped it low on the shaft. Few participants had conscious awareness of their strategy, as was true of the participants in the handle-rotation studies.

Another experiment on the grasp-height effect focused explicitly on the degree of control required at the end versus the beginning of the task. As before, university students reached out to move a plunger from a fixed height to various target heights, but the plunger was lifted from a wide or narrow ring and was then placed into a different wide or narrow ring. When the target ring had a small diameter, there was a stronger grasp-height effect than when the target ring had a large diameter, consistent with the idea that the exact grasp height mattered more when the requirement for precision at the completion of the task was high rather than low. The diameter of the start ring also affected the results: When the start ring was narrow, participants grabbed the plunger close to its base, whereas when the start ring was wide, they grabbed the plunger farther from its base. This result accords with the view that lowering one’s grasp reduces unwanted swings of the object as it is being lifted. Taken as a whole, the studies described in this section show that grasp planning is exquisitely sensitive to immediate and forthcoming demands.

Origins of Grasp Planning

Grasping an object seems so simple that we rarely think about it. Yet even casual observation of very young children shows that the skill takes time to develop. When does grasp planning begin to appear, both in individuals (ontogenetically) and, for that matter, over the course of evolution (phylogenetically)?

This question has theoretical as well as empirical implications. Being able to alter behavior in anticipation of what lies ahead reflects intelligence, albeit of a basic kind. It seems reasonable, therefore, that the phenomena described above would appear before the emergence of other, more sophisticated behaviors. Surprisingly, the tests of this prediction have not uniformly supported it.

During the first year of life, infants make remarkable strides in their ability to plan their physical actions. By four months of age, they can adjust the width of their grasps as they reach for objects of different sizes. By 10 months, of age, they can adjust the speed of their hand movements as they approach objects depending, for example, on whether they intend to toss the objects or place them in a container.

One might expect infants, and certainly young children, to be capable of second-order grasp planning. Bianca Jovanovich and Gudrun Schwarzer of the University of Giessen, Germany, showed that some toddlers as young as 18 months old do indeed exhibit second-order grasp planning. However, it takes years for most children to show such planning as commonly as adults. Young children typically grasp an object the same way regardless of what they will do with it, even when they know what the future task will be. For example, when young children use one hand to pick up a horizontal dowel whose left or right end must be brought down onto a target, they typically use a palm-down grasp regardless of whether that grasp results in a thumb-up or thumb-down posture. Not until 10 years of age on average do children prospectively alter their grasp orientations as reliably as adults do.

This is a very puzzling outcome, especially considering that typically developing children less than 10 years of age display sophisticated cognitive achievements such as speaking in linguistically complex sentences. Why don’t they grasp objects in a more adult-like way? The answer comes from an unexpected source: chimpanzees.

In 2012, Scott Frey of the University of Missouri and Daniel Povinelli of the University of Louisiana reported a study in which they gave chimps the chance to grab a rod to get food from its end. The position of the rod was such that an impulsive grab would let the chimp bring the food to its mouth, but the chimp would then have to hold the rod during feeding with a hand posture that, for us humans, is awkward. The chimps did just that, however, holding the rod in extreme joint angles while they ate what was on the end of the rod, suggesting, at least initially, that they could not carry out second-order grasp planning. However, another experiment by Frey and Povinelli ruled out this interpretation. This time, the experimenters designed a new setup such that the chimps could get food only by taking hold of the rod with an awkward (for humans) posture, after which they could turn the rod to bring its baited end to the mouth, now with the arm in a comfortable (again, for humans) posture. The chimps were able to accommodate to this setup, demonstrating their capacity for second-order grasp planning.

This pair of experiments shows that although chimps have the intelligence for second-order grasp planning, they tend to plan only as far as needed. Because chimps don’t seem to mind holding their arms in postures that human adults find uncomfortable (as shown above), they have no need to plan grasps that avoid those postures. This tolerance might also be true of young humans: If children don’t find extreme joint angles uncomfortable, and if they have little difficulty controlling their limbs at those joint angles, there would be no need for them to plan grasps that avoid those extreme postures.

We plan to test this explanation directly by surveying children for posture comfort ratings or analogous preference judgments. We expect children to report extreme joint angles as being more comfortable than adults do. We also plan to study children’s positioning abilities at various joint angles. We expect children to show fewer positioning difficulties at extreme joint angles than adults do. Even though we have not yet tested these predictions, we are cautiously optimistic about the limb-flexibility account, both because of Frey and Povinelli’s elegant studies and because, by age 10, children make the transition to puberty, when limb flexibility changes. Recall that age 10 is the average age at which children finally show second-order grasp planning as reliably as adults do.

Chimpanzees are tool users, so it makes sense that they can alter their initial grasp to anticipate the demands of later tasks insofar as tool use is a cognitively sophisticated skill. Little is known, however, about the possibility of grasp planning in other creatures that are not tool users. If in evolution the cognitive underpinnings of tool use emerged in one fell swoop, as some authors have proposed, then non-tool-users would not be expected to show second-order grasp planning. Conversely, if the cognitive basis for tool use emerged more gradually, then even non-tool-using animals might show second-order grasp planning.

A series of new studies, including work by one of us (Weiss) with Stacey Zander and Peter Judge of Bucknell University, has confirmed this prediction. The work relates to lemurs, tamarins, and squirrel monkeys, which are not tool users and have limited dexterity, as shown in the photographs at above right. Research in our labs, illustrated at right below, documented that all of these animals show second-order grasp planning. By contrast, another group of primates— capuchins and rhesus monkeys, who are tool users but are highly dexterous—do not show second-order grasp planning. This surprising finding suggests that the ability to engage in second-order grasping is better predicted by lack of dexterity than by tool use.

The fact that second-order grasp planning is seen in primates that have not developed the ability to use tools suggests the capacity to plan for tool use came from a more basic ability to plan for forthcoming actions. According to this hypothesis, the full set of cognitive competencies for tool use did not spring forth in one quantum leap. Our work suggests instead that one of the important scaffolds for tool use, second-order grasp planning, was in place as long as 65 million years ago, when the lemur line and the hominid line diverged.

The Right Hand and the Left

So far we have focused on grasp orientations and grasp heights, but these are just two of the variables that distinguish the many ways to grasp an object. Another variable is which hand to use. When an object is presented not directly in front of individuals but to their right, right-handed people (90 percent of the population) typically grab it with the right hand. When the same object is presented to the left, the same people often use the left hand. This is an example of first-order planning. But what about hand choice and second-order planning: Is the preference to use one hand or the other overridden by more futuristic concerns?

A recent study at Pennsylvania State University by Chase Coelho, Breanna Studenka, and one of us (Rosenbaum) showed that it is. The study involved asking university students to reach for a horizontal dowel whose left or right end would be set down on a target. In one condition, the participants were told which hand to use and could choose the grasp they preferred, either palm up or palm down. In the other condition, the same participants were told which grasp to use and could choose which hand to use, either left or right. When hand was specified and grasp could be chosen, the resulting grasps that were chosen usually ensured a final thumb-up position. But when grasp was specified and hand could be chosen, the chosen grasps did not ensure use of the normally preferred hand; rather, participants tended to use whichever hand ensured a final thumb-up posture. Considering that all the participants were right-handed, the fact that they chose the right and left hands equally often when grasp was specified was a stunning result, indicating that the preference to end in a thumb-up posture overrides the preference to use the right hand.

This last study, like all those described so far, involved objects that could be grasped and moved with one hand. What about objects that require two-hand grasps? Does grasping with two hands also reflect second-order planning? In a study published in 2006, Melanie Lam, Kristin McFee, and Romeo Chua of the University of British Columbia, and Daniel Weeks of Simon Fraser University showed that it does. They asked university students to carry out a variant of the rod-turning task that used a huge cardboard tube. Almost all the participants took hold of the tube in a way that let them place it in its final position with their thumbs up. This result not only suggests that second-order planning applies to the two hands; it also suggests that second-order planning applies to the entire body, for participants in this study bent at the knees and hips, among other joints, to achieve the final postures they adopted.

The capacity for whole-body second-order planning was further supported in a recent study led by William Land, then of Bielefeld University, Germany. This study showed that when university students walked up to a piece of furniture that had a drawer, they placed themselves in front of the drawer differently depending on where they knew an object was inside the drawer and how they would move it.

Given these results, one might expect comparable planning capabilities in people preparing to move two objects rather than one. To test this prediction, Matthias Weigelt and Wolfgang Prinz of the Max Planck Institute for Human Cognitive and Brain Sciences in Munich, and Wilfried Kunde, then of Martin Luther University in Halle/Saale, Germany asked university students to reach out and grasp two horizontal dowels with their two hands. Different ends of the dowels were supposed to be placed on two targets. The participants grasped the dowels in such a way that they ended the placements with both hands thumb-up.

A similar experiment by two of us (van der Wel and Rosenbaum) showed that two-handed grasps are tuned to the masses of the objects to be moved. Our participants reached for two equally tall plungers standing side by side. The task was to move the left plunger to a shelf farther to the left and to move the right plunger to a shelf farther to the right. The heights of the shelves varied, as did the masses of the plungers (which the participants had a chance to heft ahead of time). The greater the difference in the plungers’ masses, the more the participants adjusted their initial grasps to achieve symmetric final positions, that is, bilaterally symmetric (equally high) grasps at the ends of the moves. Conversely, the smaller the difference in the plungers’ masses, the more the participants adjusted their initial grasps to permit symmetric starting positions. These results indicate that second-order grasp planning is sensitive not just to kinematics but also to dynamics.

How the Brain Plans the Moves

In considering all these facts about grasp planning, we wanted to understand how the brain achieves the seemingly effortless process of planning grasps. A recent experiment by one of us (Herbort), along with Martin Butz, also of the University of Würzburg at the time, suggested an answer. The experiment had university students reach out with one hand to turn a knob either clockwise or counter-clockwise over a short, medium, or long angular distance, as illustrated on page 372. Although the direction of knob rotation had a large effect on initial grasps (which afforded easy-to-control final grasps), the amplitude of rotation had a much smaller effect.

A simple explanation of this result is that participants’ grasp choices reflected a mixture of first-order and second-order planning. When small rotations were required, grasp choices may have mainly reflected first-order planning, taking into account such factors as the position of the arm before reaching and the visual appearance of the setup. When large rotations were required, grasp choices may have mainly reflected second-order planning, using the best grasps that were learned for the clockwise or counter-clockwise rotations. When intermediate rotations were required, grasp choices may have relied on more of a mixture of first- and second-order planning.

The scientific understanding of grasp planning is still at a primitive stage.

• Facebook
• Twitter

This model has the appealing feature that it affords quick decision making without the need for mental simulation. Once a task has been practiced, as in the studies just mentioned, a response that has proven successful can be retrieved without the need for calculation de novo. Similarly, the retrieval needn’t lock one into repeating earlier responses that might prove maladaptive for new tasks. Mixing first-order and second-order planning can enable novel moves, not just probabilistically from one trial to another but also within trials, on a grasp-by-grasp basis. Within-trial mixtures leading to novel moves can be achieved if the brain sums multiple retrieved responses, assigning a weight to each response in proportion to its usefulness for the task at hand.

This approach is neurally plausible, as was demonstrated more than 20 years ago in classic research by Apostolos Georgopoulos, then of Johns Hopkins University, who studied the control of directed reaching by the brain’s motor cortex. Georgopoulos showed that the summed activation of multiple neurons, each tuned to a different reach direction, reliably predicted the direction of forthcoming reaches. The strength of a neuron’s contribution increased with the proximity of its preferred direction to the required reach direction. The same method could be used for hand rotations.

In this article we have only begun to scratch the surface of this field. We have omitted many important subjects such as the role of different brain regions, clinical issues, reaching for moving objects, carrying out more complex object manipulations (for instance, tying shoelaces), eye-hand coordination, and object manipulation in the elderly, for which there has been remarkably little research.

The elementary nature of the tasks we have studied attests to the fact that the understanding of grasp planning is still primitive. Otherwise, we would have been unable to make our initial discoveries in the way described here, often by simply observing people carrying out everyday tasks. Understanding how people perform in the everyday environment is of such obvious importance in robotics, medicine, and everyday life, and is also so fascinating to us that we plan to keep our eyes wide open for new phenomena. Along with our colleagues—many of whose work could not be covered here—we hope to answer more fully the deceptively simple question, What’s in a grasp?