American Scientist

To the Editors:

Kevin Heng (“The Nature of Scientific Proof in the Age of Simulations,” Perspective, May–June) calls attention to a number of important issues but overlooks recent work in philosophy of science on precisely those concerns. I have a couple of comments, informed by that work.?

Heng emphasizes that simulations should be falsifiable. But it is misguided to aim to falsify simulations or simulation models; we already know that our models are imperfect representations of real-world systems, as Heng himself discusses. Instead, we should aim to test (and thereby confirm or disconfirm or falsify) a model’s adequacy for a purpose of interest. Moreover, we should aspire toward tests that are particularly informative—tests that provide good evidence that a model is or is not adequate for our purpose.

Heng also suggests that, alongside experimenting and theorizing, simulation constitutes a third way of establishing scientific truths. It is worth asking: What kinds of scientific truths? Simulations are produced by solving equations; on their own, they show us (at best) the implications of a set of physical assumptions. Yet this result can be valuable. For instance, simulations can reveal that a set of physical processes—as represented by a set of equations—is sufficient to produce a phenomenon of interest; this outcome occurs when the phenomenon is produced in a simulation whose numerical methods are not in doubt. It is a further question, however, whether the real phenomenon is actually produced in the way the simulation suggests.

Wendy Parker
Durham University, UK

Dr. Heng responds:

Dr. Parker raises viewpoints that are worthy additions to the conversation. The purpose of the essay is not to dictate to the community a set of rules. I certainly do not believe that there is only one way of doing things. Rather, its purpose is to stimulate conversation on using simulations constructively. The points I raised in the essay are a reaction to bad habits I have encountered over the course of my work and travels, rather than a blanket dismissal of all things related to simulations. In fact, I am rather fond of simulations myself. Although it is true that work that is not mainly focused on falsifiability may be valuable at times (I do some of this myself), I still believe that falsifiability should be the guiding principle of a theoretical astrophysicist whenever possible. I make no such claim for other theoretical disciplines.

To the Editors:

Kevin Heng nicely elucidates the problems associated with megasimulations, which are becoming more dominant in astrophysics in particular, and the worries they raise in part of the astrophysical community.

The crucial part of this discussion, in my view, is the lack of predictability in some of the simulations. As noted by philosopher of science Karl Popper in his book The Logic of Scientific Discovery , a scientific theory must be falsifiable, and falsifiability hinges on the ability to make predictions. Simulations that can recreate known phenomena but cannot make predictions are as fun to play with as SimCity, but they also share the same scientific value.

Megasimulations are extremely powerful for advancing scientific understanding, but should be used only at a level where clear predictions can be made. Incorporating finer details in a simulation with a large set of free parameters may be a waste of time, both for the researcher and for the readers of the resulting papers. Moreover, such simulations may create the wrong impression that some problems are essentially fully solved, when in fact they are not. The inevitable subgrid physics makes the use of free parameters unavoidable. For example, using them in weather forecast modeling is the standard practice. But, the predictions of these megasimulations are tested on a daily basis and now lead to remarkably accurate weather forecasts up to a week ahead. If one cannot test his or her model by making (new) predictions, he or she will not get closer to the truth.

Ari Laor
Technion, Haifa, Israel