Paradoxical behavior can be understood in the context of evolutionary stable strategies. The trick is to discover which game the animal is playing
The science of behavioral ecology thrives on paradoxes, baffling inconsistencies between intuition and evidence that engage our attention and stimulate further investigation. Mathematical models can modify our intuition by showing that apparently sensible explanations are actually problematic, or that seemingly outrageous proposals are downright reasonable. This is especially true in the study of animal contests, where the consequences of an interaction between two animals with opposing interests are difficult to guess.
Consider a famous example known as the "handicap principle." The behavioral ecologist Amotz Zahavi of Tel Aviv University argued that animals with conflicting interests should evolve behavioral displays that are costly to the signaler, even if they lower its chances for survival. By showing that it can endure a handicap, the animal reliably indicates its high quality, a message that other animals do well to respect. For example, on sighting a predator, a gazelle may stott—that is, jump high in the air on all four legs—several times before fleeing, thereby demonstrating that it is in superb physical condition, and that the predator would only waste time and energy by pursuing it. This hypothesis was initially rejected by many partly because it contradicted the biologist's intuition that evolution should favor signals with low costs to the animal producing them, especially since these costly behaviors will be passed onto the signaler's offspring.
Yet formal models of communication revealed that the handicap principle is logically sound under certain conditions. In particular, the magnitude of the handicap must increase with the intensity of the signal, and the cost must be especially damaging for animals of lower quality. For example, if a weak gazelle stotted as vigorously as a strong one, then it would waste what little strength it had and be unable to flee from a pursuing predator. These models have convinced many biologists that a previously unaccepted idea has broad explanatory power.
Efforts to resolve such questions in animal behavior have relied increasingly on collaborations between biologists and mathematicians, using analytical tools called games. In particular, behavioral ecologists use evolutionary games. A game in this context is a mathematical model of strategic interaction, which arises when the outcome of an individual's actions depends on the actions of others.
A game has three components. First, there are at least two interacting individuals, called players. In an evolutionary game played within a single species, the set of players is an ecotype, a population of animals in a given ecological environment. For example, a population of spiders in a grassland habitat and the same species in a riparian habitat form two different ecotypes.
Second, each player has a set of feasible strategies. In an evolutionary game, this set is the same for every player and is constrained by the information structure of the interaction. For example, animals can modify their behavior in different circumstances, such as whether they are owners or intruders, only if they are aware of such roles.
Third, the pattern of interaction must be well defined and accompanied by a formula for how each player's reward from the interaction depends on its strategy and on those of the other players. In evolutionary games, the rewards are measured in terms of expected future reproductive success.
For a game to be useful, it must be possible to identify one or more strategies from among those feasible as the "solution" for a given purpose. In our case, the solution is the behavior that can be expected to evolve by natural selection. If a behavior is fixed in a real population, then it must at least be true that every feasible alternative behavior would yield a lower reward, for otherwise the alternative behavior would have spread into the population. The relevant solution concept, which was introduced a quarter of a century ago by John Maynard Smith of the University of Sussex, is that of an evolutionary stable strategy, a population strategy that yields a higher reward than any feasible mutant strategy (that is, any newly introduced alternative strategy).
Just as a model population is only a caricature of a real population, so a paradox is only a caricature of real ignorance. So, in terms of realism, a game and a paradox are a perfect match. A game theorist strives to unravel the paradox by establishing conditions for an evolutionary stable strategy to exist in the model population, and by analyzing its properties when it does exist. Now, a paradox arises because evidence fails to support intuition, which (assuming the evidence to be sound) can happen only if the intuition relies on a false assumption, albeit an implicit one. So the way to resolve a paradox is to spot the false assumption.
In other words, if a paradox of animal behavior exists, then we have wrongly guessed which game best models how the real population interacts, and to resolve this paradox we must guess again—if necessary, repeatedly—until eventually we guess correctly. Assuming the validity of our solution concept, that is, assuming that the observed behavior corresponds to some evolutionary stable strategy, there are only three things we could be wrong about: the ecotype, the information structure and strategy set or the pattern of interaction and reward. So there are also only three things to be right about. Each can be important, as we illustrate with examples from our recent work on animal contests among damselflies, mantis shrimps and spiders.