False-Positive Psychology

Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant

Summary: In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

Joseph P. Simmons,Leif D. Nelson and Uri Simonsohn
@ False-Positive Psychology:


Cf. The CRIIGEN approach as discussed by Hammond et al 2012

1. Experimental designThe authors of this study assert that it was conducted in a GLP environment and according to OECD guidelines. They did not follow OECD GLP guidelines nor OECD testing guideline (TG) 453 for conduct of a combined chronic toxicity/carcinogenicity study. OECD GLP’s require “Detailed information on the experimental design, including a description of the chronological procedure [e.g., start date, end date] of the study, all methods, materials and conditions, type and frequency of analysis, measurements, observations and examinations to be performed, and statistical methods to be used (if any)” and … “The study should be conducted in accordance with the study plan”. Apparently, the authors’ original intent was not to conduct a carcinogenicity study “…we had no reason to settle at first for a carcinogenicity protocol using 50 rats per group.” (Seralini et al., 2012), but at some point during the in-life phase, they changed the purpose of the study by extending it for 2 years to assess potential carcinogenicity. Assuming they had a protocol at the start of the study, they did not follow it as they substantially altered the purpose and the design of the study while it was in progress. This should be considered a violation of GLP guidelines as the study was not conducted in accordance with the original study plan. If they wanted to carry out a carcinogenicity study, they should have terminated the existing study, and prepared a new study plan adapted from OECD TG 453. They did recognize, as stated above, that they needed a larger number of animals (a minimum of 50 rats/sex/group) for a carcinogenicity study, instead of the 10 rats/sex/group that they had in their existing study. For reasons which will be discussed later, their study did not have enough animals to draw any meaningful conclusions.

The remedies suggested by Simmons et al 2011 are:

Requirements for authors

We propose the following six requirements for authors.

1. Authors must decide the rule for terminating data collection before data collection begins and report this rule in the article.
2. Authors must collect at least 20 observations per cell or else provide a compelling cost-of-data-collection justification. .
3. Authors must list all variables collected in a study. 
4. Authors must report all experimental conditions, including failed manipulations. 
5. If observations are eliminated, authors must also report what the statistical results are if those observations are included. 
6. If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate. 

Guidelines for reviewers

We propose the following four guidelines for reviewers.

1. Reviewers should ensure that authors follow the requirements. 
2. Reviewers should be more tolerant of imperfections in results. 
3. Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions. 
4. If justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication. 

See also other Pundit posts

• GMO Statistics Part 24. Correlations without Causation.
• GMO Statistics Part 17. Correlation is not causation, even for chocolate eating Swedes