American Scientist

Charles Darwin’s On the Origin of Species laid the foundations for evolutionary biology and understanding of life on Earth. Even as he was marshalling evidence for the theory of natural selection, Darwin made a point of highlighting observations that seemed to challenge his ideas:

I… will confine myself to one special difficulty, which at first appeared to me insuperable, and actually fatal to my whole theory, I allude to the neuters or sterile females in insect communities: for these neuters often differ widely in instinct and structure from both the male and fertile females, and yet from being sterile they cannot propagate their kind.

In the insect societies that Darwin was alluding to, such as ants or termites, there is a reproductive division of labor: Some individuals forgo their own reproduction and help others reproduce. To express Darwin’s special difficulty in modern terms, these societies are challenging to understand because it’s not immediately apparent how natural selection can preserve the genes that underlie nonbreeding and helping behaviors. Evolutionary biologists have puzzled over such cooperative behaviors ever since Darwin highlighted the challenge that they present to the theory of natural selection.

Although the social insects present extreme cases of sociality, social birds and mammals exhibit similar, if less extreme, forms of sociality. Indeed, behavioral ecologist Paul W. Sherman of Cornell University and others have suggested that the difference between social insects and social vertebrates is only one of degree, the various forms of sociality lying on a continuum. For example, birds such as white-fronted bee-eaters (Merops bullockoides) and mammals such as naked mole rats (Heterocephalus glaber), also live in complex societies in which some individuals forgo reproduction and help others to reproduce, at least at some point in their lives. Anthropologist Sarah B. Hrdy of University of California, Davis, and others have even argued that such cooperative breeding may have been pivotal in human evolution—essential to support our long and unusual life histories.

Social vertebrates have proven invaluable for developing and testing theories of social evolution—they often exhibit a high level of flexibility in behavior, which enables researchers to manipulate key variables and measure individual responses. In the 1980s, Stephen T. Emlen of Cornell University presented an evolutionary framework for understanding nonbreeding and helping strategies, based on studies of cooperatively breeding birds and mammals (see the sidebar “An Evolutionary Framework for Cooperative Behavior” below). This framework still guides much of today’s research. Emlen emphasized that, to understand cooperative breeding, there are two main questions to answer: First, why do individuals help; second, why don’t individuals disperse to breed elsewhere? The first question focuses on the reproductive payoff that individuals accrue from their current actions, whereas the second focuses on the payoff associated with alternative actions. This framework focused attention on two major reasons that nonbreeding and helping behaviors would evolve: kin selection and ecological constraints.

Complex Societies of Coral Reef Fish

Because this evolutionary framework for understanding cooperative breeding was developed based on studies of birds and mammals, it was unclear whether it extended more broadly to other groups of animals. To find out, we have spent the better part of two decades studying the complex societies of two coral reef fishes: the clown anemonefish (Amphiprion percula), in Madang Lagoon, Papua New Guinea, and the emerald goby (Paragobiodon xanthosomus), at Lizard Island, Australia.

The societies of these fishes bear a striking resemblance to the complex societies of birds and mammals. In both species, groups of individuals are found in close association with invertebrate hosts (anemones or corals) that provide the fish with protection from predators. Each host contains one group of fish, which is composed of a breeding pair and a small number of nonbreeders. Within each group there is a size-based dominance hierarchy: The breeders are the largest individuals, and the nonbreeders get progressively smaller. These fishes, like many coral reef fishes, are hermaphroditic: Clown anemonefish can change sex from male to female; emerald gobies can change sex from female to male.

The aim of our work has been to use these fishes to test the robustness of the current framework for understanding the evolution of cooperative breeding and generate new insights into social evolution. Here, based on our work and that of others, we present a simple but expanded evolutionary framework that captures two other major reasons that nonbreeding and helping behaviors would evolve.

Kin Selection in Birds and Mammals

The theory of kin selection, formalized by the evolutionary biologist William D. Hamilton in the 1960s, emphasizes that there are two ways for individuals to pass their genes to the next generation—either directly, by producing their own offspring, or indirectly, by enhancing the offspring production of their relatives. Helping relatives can be favored by kin selection because the relatives’ offspring share copies of the helper’s genes. This hypothesis makes two critical predictions: First, helpers enhance the fitness of breeders; and second, helpers are closely related to breeders.

In the late 1980s, studies of white-fronted bee-eaters, a colonially nesting bird found in Kenya and Tanzania, provided support for both of these predictions: Each additional helper results in the breeders raising, on average, half a chick more to fledging, and helpers tend to be the offspring of the breeders they help. There is now a large body of evidence showing that helpers enhance the fitness of close kin and, consequently, that kin selection helps to explain many of the cases of helping behavior observed in social birds and mammals.

Ecological Constraints in Vertebrates

The theory of ecological constraints, formalized by Emlen, emphasizes that two alternative options are available to individuals—either they can disperse to breed elsewhere or they can stay on their natal territory as nonbreeders and help rear siblings. Many factors can be encompassed by the umbrella term ecological constraints: Staying can be favored if the habitat is saturated, meaning there are no high-quality habitat vacancies, or because it is dangerous to move between patches of suitable habitat. This hypothesis makes three predictions: First, individuals will stay as nonbreeders when there is some ecological constraint; second, critically, the likelihood of individuals dispersing to breed elsewhere will increase when the ecological constraint is relaxed; and, third, competition for breeding positions will be intense when habitat vacancies arise.

In the late 1980s and early 1990s, studies of naked mole rats, a subterranean rodent found in Ethiopia, Somalia, and Kenya, provided observational support for the first and third predictions: Unpredictable food availability (large plant tubers) and hard arid soils reduce the reproductive payoff associated with dispersing, and competition for breeding vacancies following the death of a breeder is intense and sometimes bloody. Subsequent studies have yielded considerable observational and experimental evidence for all three predictions, demonstrating that ecological constraints help explain many cases of nonbreeding. In birds and mammals, nonbreeding and helping behaviors often go hand in hand, because when an individual’s independent breeding options are poor it can stay home and help its relatives reproduce.

Kin Selection in Coral Reef Fishes

Guided by the evolutionary framework, we began by investigating whether clownfish nonbreeders might forgo their own reproduction and help the breeders reproduce, thereby gaining indirect genetic benefits.

To test the predictions of this hypothesis, we spent a year in Papua New Guinea, scuba diving every day, monitoring survival and reproduction in 71 groups of clownfish on two reefs. (This was as amazing as it sounds, but bouts of malaria and strange fungal infections mean that the experience is not for the faint-hearted). We complemented this monitoring with an experiment in which we removed all nonbreeders from 14 of the 71 groups and examined the effect of the removal on the fitness of breeders (see the schematic at right).

After another couple of years spent compiling the data and conducting statistical analyses, we were faced with an intriguing result: The nonbreeders had no effect on the survival or reproduction of breeders. Further, genetic analyses showed that the nonbreeders were not closely related to the breeders, because they disperse from their natal territories as larvae very early in life. Taken together these null results indicated something quite remarkable: Kin selection, one of the founding concepts of social evolution, would play no role in explaining these fishy societies.

Future Selection in Coral Reef Fishes

If nonbreeders are not gaining indirect genetic benefits, then they must be contributing genetically to the next generation in some other way—otherwise their behavior would be weeded out by natural selection. In the late 1970s and early 1980s, based on their studies of the Florida scrub jay, the ornithologists Glen E. Woolfenden and John W. Fitzpatrick emphasized the possibility that nonbreeders might accrue direct benefits in the future, by inheriting the territory following the death of the breeders. The idea can be formalized in a manner directly analogous to kin selection (see the sidebar “A Modified Framework for Cooperative Behavior” below).

To determine whether the behavior of clownfish was favored by future selection, we tested two critical predictions: First, the largest nonbreeder from an anemone would inherit the territory when a dominant breeder died and left a breeding vacancy; and second, these breeding vacancies would not be usurped either by smaller nonbreeders from the anemone or by nonbreeders from elsewhere.

We tested these predictions by monitoring 57 of the 71 groups and recording which individuals from the population filled breeding vacancies when they arose. We complemented these observations with an experiment in which we removed breeding males from 16 groups on a third reef, and determined which individual from the 26 groups on that reef filled the breeding vacancy created (see the schematic at right).

The results were unambiguous: In all cases the largest nonbreeder from the anemone inherited the breeding vacancy; in no case did the smaller individual from the anemone, or a nonbreeder from elsewhere, usurp the vacancy. Simply put, individuals form a perfect queue for breeding positions. These results are one of the clearest demonstrations (not confounded by kin selection) that individuals will adopt nonbreeding positions solely because of the potential for reproductive success in the future.

Ecological Constraints in Fishes

The benefits that nonbreeders gain from inheriting territories in the future are necessary but not sufficient to explain their behavior. A complete understanding requires that we also explain why nonbreeders don’t disperse to breed elsewhere.

To determine whether their behavior was favored because of ecological constraints we tested two critical predictions: First, the likelihood of dispersal would increase as the availability of suitable habitat increased; second, the likelihood of dispersal would decrease as the risks of movement increased. We tested whether dispersal occurred when habitat became available by monitoring the 97 groups of clownfish, after removing males or nonbreeders from 30 of them to create habitat vacancies. The manipulated groups could be as little as one meter or as much as 100 meters from their neighbors. In clownfish, no individuals dispersed between anemones even when habitat saturation was reduced in this manner.

More compellingly, we complemented this with an experiment using emerald gobies, in which we manipulated habitat saturation and risks of movement using 31 pairs of coral habitats (see the schematic above). To move between corals, gobies must risk being eaten, and that risk increases with distance between corals. In emerald gobies, the likelihood of dispersal increased as the availability of suitable habitat increased, whereas it decreased as risks of movement increased. Indeed, in these fish, the probability of nonbreeders moving between experimental corals is effectively zero when they are just one meter apart, and corals are much further apart than this in nature.

Taken together, these results indicate that ecological constraints play an important role in the evolution of nonbreeding in coral reef fishes, just as they do in the evolution of nonbreeding in birds and mammals. In coral reef fishes, however, it is nonbreeding and queuing that tend to go hand in hand (rather than nonbreeding and helping), because when an individual’s independent breeding options are poor its only way to reproduce is to settle in a territory as a nonbreeder and wait to inherit a breeding position.

Social Constraints in Coral Reef Fish

Future benefits and ecological constraints are sufficient to explain why individuals wait to inherit a breeding position rather than disperse to breed elsewhere. However, they do not provide the answer to another perplexing question: Why do individuals wait to inherit a breeding position rather than contest for a breeding position? One possibility is that individuals don’t contest for breeding positions because of social constraints (a concept directly analogous to the concept of ecological constraints), which might lower the reproductive payoff associated with alternative actions within the group (see the sidebar “A New Evolutionary Framework for Cooperative Behavior” below).

An example of a social constraint would be the potential for inbreeding, emphasized by the behavioral ecologists Walter A. Koenig and Frank A. Pitelka, both then at University of California, Berkeley, based on their studies of acorn woodpeckers (Melanerpes formicivorus), which live in family groups. If the only breeding positions that can be obtained by contesting would involve mating with relatives, and inbreeding has a low reproductive payoff, then selection may favor waiting to breed with a nonrelative rather than contesting to breed with a relative. The potential for inbreeding is just one factor that can be considered under the umbrella term social constraints, but there might be many types of social constraint, just as there are many types of ecological constraint.

To determine whether the queuing behavior of clown anemonefish and emerald gobies might be favored because of social constraints, we began by thinking about what those constraints might be. During our yearlong field study of clown anemonefish we observed that dominant individuals occasionally evicted their immediate subordinates when the size difference between the two individuals was small. Further, we found that the size ratios between individuals adjacent in rank were not random. Rather, well-defined size ratios occurred between individuals adjacent in rank, and these size ratios seemed to be maintained by subordinates regulating their growth (see graph above).

The idea that a vertebrate could regulate its growth in response to social context was flabbergasting and took (and still takes) many researchers in the field by surprise. Even now, we don’t understand exactly how the subordinates regulate their growth, although it seems to involve a self-imposed reduction in food intake that one might call dieting or fasting, at least in the emerald gobies. The extent to which social birds and mammals adaptively modify their food intake, growth, and size in response to social context is unknown, although interesting patterns have been documented in mole rats, meerkats, and humans.

Putting the observations of eviction and well-defined size ratios together, we formed three complementary hypotheses. First, selection would favor dominant individuals that evicted rather than tolerated subordinates who became too close in size—otherwise the dominant would risk being overthrown. Second, because of the threat of eviction, selection would favor subordinate individuals that regulate their growth and size. Third, because of the size differences maintained, selection would favor subordinate individuals that wait to inherit rather than contest for a breeding position.

To determine whether the social constraint existed and whether the observed subordinate behavior was favored because of it, we used emerald gobies to test a critical prediction of each of these hypotheses: First, the likelihood of dominants evicting subordinates would increase as they became more similar in size; second, subordinates would maintain a size difference with respect to their immediate dominant that did not elicit eviction; and third, the likelihood of a subordinate ascending in rank by queuing would be greater than the likelihood of ascending in rank by contesting.

We tested these predictions by staging 56 contests between pairs of nonbreeders. Pairs of contestants were chosen such that they fell into one of four categories (size ratio ≈ 0.85; size ratio ≈ 0.90; size ratio ≈ 0.95; size ratio ≈ 1.0), where the size ratio is given by the size of the smaller individual (the subordinate) divided by the size of the larger individual (the dominant). The paired contestants were released into a coral in which neither had prior residence. The outcome of each staged contest was scored as subordinate evicted, dominant evicted, or neither evicted.

In support of the first prediction, we found that dominants were more likely to evict their subordinates as the size ratio of the contestants tended toward 1.0. In support of the second prediction, we found that the probability of subordinate eviction was minimized when the pair’s size ratio matched that which is maintained under natural conditions (0.93). Finally, given the size ratio maintained under natural conditions, the probability of a subordinate ascending in rank by winning a contest is effectively zero, whereas the probability of ascending in rank by outliving at least one of its dominants is at least 0.66, assuming mortality rates are independent of rank (see the graph above).

Taken together, these results are one of the clearest demonstrations that individuals will adopt nonbreeding strategies within a group because of a social constraint—in this case, the hidden threat of eviction. Since we published our findings, zoologists Tzo Zen Ang and Andrea Manica, both at University of Cambridge at the time, have reported similar results in other coral reef fish societies. The role of social constraints in animal cooperation, particularly the role of hidden threats, is now an active and exciting area of research.

A New Evolutionary Framework

Darwin highlighted the challenge that the nonbreeding and helping behaviors observed in insect societies presented for his theory of natural selection and, since then, generations of biologists have set about rigorously testing alternative hypotheses for why such cooperative behaviors might be favored by natural selection.

From the mid-1960s through the mid-1990s, behavioral ecologists developed a framework for understanding the evolution of nonbreeding and helping behavior. That framework emphasized that individuals might cooperate because their behavior provides indirect genetic benefits, and that individuals might not pursue alternative, noncooperative options outside of the group because of ecological constraints. That framework emerged from studies of cooperatively breeding birds and mammals, leaving us to wonder how things would play out in other parts of the animal kingdom.

Our goal has been to test the generality of this evolutionary framework and to generate new insights by studying complex societies of coral reef fishes. Our studies, along with those of many others, have enabled us to expand this framework. This framework emphasizes that individuals might cooperate because their behavior provides future genetic benefits, and that individuals might not pursue alternative, noncooperative options inside the group because of social constraints (see the sidebar “A New Evolutionary Framework for Cooperative Behavior” above).

For natural selection to favor the evolution of cooperative behavior via kin selection or future selection, individuals must be able to assess or enhance the probability of accruing the benefits of cooperation. This statement does not mean that individuals have to know the probability of sharing an allele identical by descent or the probability of an event occurring in the future. Rather, individuals must have a simple rule—a kin recognition system or future recognition system—that enables them to assess these probabilities with a reasonable degree of certainty and then act accordingly. There has been a lot of work on kin recognition systems; work on future recognition systems is in its infancy and is a potentially fascinating avenue of research.

We believe that expanding the evolutionary framework to encompass four key concepts—kin selection, future selection, ecological constraints, and social constraints—provides a more complete understanding of the diversity of nonbreeding and helping strategies observed in complex animal societies. Also, this expanded view reveals a path toward unification of studies of cooperation across the biological and social sciences.

The evolutionary framework for understanding cooperation presented here has a striking parallel in economic bargaining theory. An excerpt from University of Warwick economist Abhinay Muthoo’s introduction to bargaining theory helps illustrate this point:

Two main determinants of such (marital) negotiations are each individual’s outside and inside options. Their outside options are the payoffs that they obtain from divorce, which might, for example, be their payoffs from being single, or from finding an alternative partner. Their inside options are their payoffs from remaining married but with generally uncooperative behavior (such as constant fights and arguments…).

In our expanded evolutionary framework, the payoff associated with dispersing to breed elsewhere, which is influenced by ecological constraints, is directly analogous to the outside option in bargaining theory—the payoff associated with leaving a cooperative interaction. Similarly, the payoff associated with contesting to breed within the group, which is influenced by social constraints, is directly analogous to the inside option in bargaining theory—the payoff derived from behaving uncooperatively within an interaction.

Framed in the language of bargaining theory then, our expanded evolutionary framework becomes: Selection will favor individuals that engage in cooperative actions if the payoff associated with cooperation (which depends on direct, indirect, and future genetic benefits) is greater than the payoff associated with either the noncooperative outside option (which depends on ecological constraints) or the noncooperative inside option (which depends on social constraints). Although the nonbreeding and helping behaviors observed in animal societies seem paradoxical, they make sense when one considers the alternative options available to individuals.