Pages

Saturday, November 16, 2013

GMO Statistics Part 40. Starting with a sow's ear, a standard statistical test will not magically generate a beautiful silk purse


The Pundit's thoughts:

Simple, easily used practical tests -- medical, scientific or statistical -- don't always give the correct answer every time, as if finding the truth was an automatic mechanical algorithm.

Context matters. 

Good scientific knowledge and skill are first needed to find useful conjectures about what's happening in the complex real world, which then can be evaluated in light of results that emerge from experiments or tests based on real world events.

Given a sows ear, a standard statistical test will not magically generate a silk purse -- in the sense of serving up a proven golden truth cleanly squeezed out of the necessarily imperfect real world of observations about nature. 


Made about this: article

Are all significant P values created equal? The analogy between [medical] diagnostic tests and clinical research. (pdf file)  

Just as [medical] diagnostic tests are most helpful in light of the clinical presentation, statistical tests are most useful in the context of scientific knowledge. Knowing the specificity and sensitivity of a diagnostic test is necessary, but insufficient: the clinician must also estimate the prior probability of the disease.

In the same way, knowing the P value and power, or the confidence interval, for the results of a research study is necessary but insufficient: the reader must estimate the prior probability that the research hypothesis is true.

Just as a positive diagnostic test does not mean that a patient has the disease, especially if the clinical picture suggests otherwise, a significant P value does not mean that a research hypothesis is correct, especially if it is inconsistent with current knowledge. Powerful studies are like sensitive tests in that they can be especially useful when the results are negative.

Very low P values are like very specific tests; both result in few false-positive results due to chance. This Bayesian approach can clarify much of the confusion surrounding the use and interpretation of statistical tests.  PMID: 3573245


Browner WS, Newman TB. 
 JAMA. 1987 May 8;257(18):2459-63. 

No comments:

Post a Comment