Do We Really Need the S-word?
The use of “significance” in reporting statistical results is fraught with problems—but they could be solved with a simple change in practice
New Twist, Old Debate
The dangers of implementing arbitrary p-value cutoffs and the importance of distinguishing between statistical significance and practical significance are now well recognized by scientists. Andrew Gelman and Hal Stern address the common error of making comparisons based on statistical significance in a 2006 paper for The American Statistician, titled “The difference between ‘significant’ and ‘not significant’ is not itself statistically significant.” I contend that in addition, our tendency to repeatedly use the same wording to explain statistical results masks potential improvements in inference corresponding to recognition of these issues. My goal here is not to revisit ideas that have been described eloquently and repeatedly over the past decades (by James O. Berger and Donald A. Berry in a 1988 article for American Scientist, and by Jacob Cohen in a 1994 article for American Psychologist, among others) but instead to propose a simple change in wording. I suggest we replace the word significant with a concise and defensible description of what we actually mean to evoke by using it.
For example, if we mean that the two-sided p-value is less than 0.05, then let us say just that. Let the reader judge whether that result is to be deemed significant by our modern scale and by their knowledge of the science. If, instead, we mean the result is in fact practically important, let’s say as much and clearly communicate our justification for doing so. If we believe our research is important, let us convince our critics through our choice of other words. We can use the p-value to back up our arguments, without appealing to the s-word. If we do not feel forced to label results associated with larger p-values “nonsignificant” or “insignificant,” it may even help curb the long-standing publication bias. By replacing the s-word with defensible statements, we can easily and quickly clean up the often sloppy dissemination of scientific results.