A post at Academic Productivity describes a new (to me) problem with statistical significance: the differences in significance between studies need not themselves be significant. In other words, even if Study A finds a statistically significant effect of Treatment A but Study B finds no statistically significant effect of Treatment B, it does not follow that Treatment A is itself statistically significantly different that Treatment B. This is important since both the research community and general public want to conclude the opposite.
In my listing, this is misuse #5. Here are the others.
- Statistical significance is different than policy significance (i.e., the “who cares” problem).
- Statistical significance is often nothing more than a measurement of sample size (i.e., larger samples have lower error variance).
- Statistical significance levels are arbitrary.
- The significant-not significant dichotomy favors the null hypothesis.
- Different significances need not show significant differences.
Of course, there is also the issue of selection bias in publications. As noted in this post from Economist’s View, an unexpectedly large number of political science publications have results that are just barely statistically significant. As the post suggests, one wonders how many of those publications are second, third, or higher attempts at finding significance.
Here is the paper’s citation.
Gelman, Andrew and Stern, Hal (2006) “The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant.” The American Statistician 60 (4) (November) pp. 1–4. http://www.stat.columbia.edu/%7Egelman/research/published/signif4.pdf
