As part of their quantitative methods research training, all students of social sciences learn the theory and methodology of testing statistical hypotheses. The concept of statistical significance is an important tool for the analysis of sample-based evidence.

But statistical significance is not the same as substantive significance: hypothesis testing is often misapplied – surprisingly often in fact. Many research findings are reported as ‘significant’ without it being clear in what sense. Some results are significant in the statistical sense while of little or negligible value as social science. On the other hand, important effect sizes can be wrongly disregarded because lacking statistical significance.

I will argue that as social scientists our primary concern in statistical work is to discover effect sizes, not merely statistically significant effects. This is a major issue in research across the social science from economics, to psychology. And one on which there is quite a lot of literature.

Reading:

Cohen, Jacob, Statistical Power Analsysis, 1988.

Gigerenzer, Gerd, “Mindless Statistics”, The Journal of Socio-Economics 33 (2004) 587–606.

Gill, Jeff, “The insignificance of null hypothesis significance testing”, Political Research Quarterly, 52(3), Sep 1999, 647-674.

Kline, Rex B, Beyond Significance Testing: Reforming Data Analysis Methods in Behavioural Research, American Psychological Association, 2005.

Morrison, D.E., Henkel, R.E., The Significance Test Controversy. Aldine, Chicago, second edition 2006.

Ziliak, Stephen T. and Deirdre McCloskey, The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives, University of Michigan Press, 2008.

---------------------------------------“The standard error of regressions”, Journal of Economic Literature, XXXIV (March 1996), 97-114.

--------------------------------------- “Size matters: the standard error of regressions in the American Economic Review”, The Journal of Socio-Economics 33 (2004) 527–546.

Ziliak, Stephen T., “Guinnessometrics: The Economic Foundation of ‘Student’s’ t”, Journal of Economic Perspectives, 22(4) (2008), 199-216.