CRIIGEN's track record on statistics: What the EFSA said in 2007

EFSA review of statistical analyses conducted for the assessment of the MON 863 90-day rat feeding study, June 2007
From the Executive Summary
General
In 2004 the EFSA GMO Panel gave its opinion (EFSA, 2004a,b) on the safety of MON 863 maize for import and processing, and released a statement on the safety of MON 863 shortly after (EFSA, 2004c). The EFSA GMO Panel based its opinions and statement on a wide range of evidence, which included data from a 90-day rat feeding study. This study was performed by Covance Laboratories, in compliance with internationally agreed GLP principles (OECD, 1998), for the applicant Monsanto.
Subsequently Hammond et al. (2006) published a scientific paper based on the Monsanto report (2002), which provided fewer details than the original Monsanto report.
In a re-analysis of the MON 863 90-day rat study Séralini et al. (2007) claimed to have revealed significant variations in growth for both genders, as well as signs of hepatorenal toxicity in rats, and alleged that it cannot be concluded that MON 863 maize is a safe product.
The European Commission (DG SANCO, 2007) asked EFSA what impact the analysis performed by Séralini et al. (2007) study might have on its earlier opinion on MON 863 maize.
This EFSA report presents an assessment of the statistical methodology as applied by Monsanto (2002) and Séralini et al. (2007). It takes account of contributions from Member States, in particular the reports provided by the Commission du Génie Biomoléculaire of France (CGB) and information provided by the Agence Française de la Sécurité Sanitaire des Aliments of France (AFSSA).
The experimental design in the Monsanto study, adapted from the OECD Guideline 408 to make it more fit for the purpose of assessing GMOs, involved three factors, each at two levels: genotype(MON 863 and non-transgenic control LH82 x A634 maize); gender (Male and Female); and dose (11% and 33% level of MON 863 in the diet for the GM-fed rats). In addition to the near-isogenic control diets, rats were fed diets from six non-GMO reference lines, i.e. commercial hybrid maize genotypes.
Previous studies of these data have involved two separate statistical analyses: (1) body weight, body weight gain and food consumption; and (2) hematology, clinical-chemistry and urinalysis parameters, histopathology and organ weights.

Body weight development
...In the EFSA analysis the following points emerged: (1) there was no significant difference in body weight over the 14-week period between GMO-fed and control rats when data were averaged over the genders and doses; (2) there was no significant effect of dose (11% versus 33%) or any interaction between dose and any other factor; (3) in all weeks except for males at 12 weeks, differences in body weights between GMO and control were consistent with random variation; (4) body weights of the animals fed GM maize and control maize were within the variations found in the additional study groups fed a range of non-GM maize commercial varieties.

In conclusion, the assumptions underlying the statistical tests performed by Séralini et al. (2007) did not hold, and so their tests tended to detect more significant results than analyses based on more robust techniques. The weekly growth data showed transient differences within the study period, but no overall differences in weight could be demonstrated. Séralini et al. (2007) reported that food consumption between GMO-fed and control rats were not noticeably different, but as shown by Monsanto (2002) and further illustrated by AFSSA (2007), transient changes in food intake were the most likely source of the transient differences in body weights.

Hematology, clinical-chemistry and urinalysis parameters, histopathology
and organ weights
Séralini et al. (2007) compared their results with those of Hammond et al. (2006) which reported a less comprehensive range of data and analyses. Séralini et al. (2007) did not compare their results to those of the study reported in the Monsanto application (2002) that the GMO Panel assessed. Séralini et al. (2007) imposed a threshold to exclude potentially incidental differences, which accords with the distinction stressed by the GMO Panel (EFSA 2004 a,b) between statistical and biological significance. 479 endpoints were comparable between the studies of Séralini et al. (2007) and Monsanto (2002). Séralini et al. (2007) and Monsanto (2002) both reported significant differences between GMO-fed and control-fed rats in the same 25 endpoints and these represent the statistically most robust results.

In addition, Monsanto (2002) reported a further 10 significant differences not reported by Séralini et al. (2007); whilst Séralini et al. (2007) reported a further 13 significant differences not reported by Monsanto (2002). Differences between the two analyses were caused by the use of different variance estimates. Neither method is considered greatly superior to the other.

Séralini et al. (2007) found 40 significant differences out of 494 tested endpoints and claimed that only 25 would be expected by chance alone. However, this statement is correct only if: (1) the endpoints are independent and uncorrelated and (2) there are absolutely no systematic differences between GMO-fed and control-fed rats.

In both the Séralini and Monsanto analyses tests were performed assuming that no correlation existed between endpoints. However, the EFSA analysis reposted here shows that sufficient correlation was present to demonstrate that the probability of 40 significant differences by chance alone was not negligible. Furthermore, this analysis shows that, given the fact that variable means from different genotypes may sometimes actually be slightly different under a substantial equivalence argument, the probability that 40 positive results in a set of 494 tests were obtained by chance alone is substantial.

EFSA investigated the variability in the data in detail. This took into account variation observed from both within and between maize variety studies and animal test populations. Toxicologists use the variability from such feeding studies to aid the interpretation of statistical significance tests. This approach provides a biological context for any statistical differences found between genotypes. For the majority of endpoints the variability between GMO-fed rats and those fed the near-isogenic control was considerably smaller than that shown by the six reference maize lines.

Furthermore, this EFSA study emphasises that statistically significant effects must be evaluated with respect to their biological significance (EFSA 2004 a,b). Finally those statistically significant differences that were found did not show consistency of patterning over dose and gender.

In conclusion, the results reported by Séralini et al. on the biochemical parameters, clinical pathology and organ weights were largely consistent with the findings previously assessed by the GMO Panel and reported in the Monsanto application.

Complete pdf files for reference:

• EFSA Report on MON863.
• EFSA Report on MON863. Appendix 1.
• EFSA Report on MON863. Appendix 2 Intereview with Dr Seralini.
• EFSA Report on MON863. Appendix 3
• EFSA Report on MON863. Appendix 4
• EFSA Report on MON863. Appendix 5

An extended article collection on CRIIGEN is here (accessible also via the hotlink at the top of this blog).