On recent days I got into an online discussion with an anti-GMO activist about the recent pig study that supposedly found that genetically modified soy and corn had a negative effect:
Judy A. Carman, Howard R. Vlieger, Larry J. Ver Steeg, Verlyn E. Sneller, Garth W. Robinson, Catherine A. Clinch-Jones, Julie I. Haynes, & John W. Edwards. 2013. A Long-term Toxicology Study On Pigs Fed A Combined Genetically Modified (gm) Soy And Gm Maize Diet. Journal of Organic Systems 8: 38-54. (Abstract | PDF)
Many have destroyed their arguments (for example here, here, and here). The activist that I got into a discussion with, sent a long email with the replies from the principal author and some supporting press articles. I replied because it showed that there are in fact many problems with the paper.
Below, I copy the full replies to the criticism that I answered, with my comments in bold. You can see the full Carman’s posts at the website http://gmojudycarman.org
A SPECIFIC REPLY TO MARK LYNAS
Prominent pro-GM activist, Mark Lynas has, as expected, attacked the study by Dr Judy Carman and her colleagues for their recent work titled, ?A long-term toxicology study on pigs fed a combined genetically modified (GM) soy and GM maize diet.?
ML: The authors are GM activists/campaigners and their results shouldn?t be trusted.
Answer Summary: The authors are not GM activists; they are highly credentialed experts.
Detailed Answer: Two authors are Associate Professors in Health and the Environment, School of the Environment, Flinders University in South Australia. Another is a Senior Lecturer at Adelaide University in South Australia. Two are veterinarians, one is a medical doctor, and two are farm experts. The authors have over 60 years of combined experience and expertise in medicine, animal husbandry, animal nutrition, animal health, veterinary science, biochemistry, toxicology, medical research, histology, risk assessment, epidemiology and statistics.
The principal author lists several publications that are anti-GM, including a chapter titled “Is GM food safe to eat?” at http://gmojudycarman.org/
In addition, Carman lists on the website “Dr Carman’s government submissions” that are anti-GM (http://gmojudycarman.org/
ML: Funding for the research was derived from anti-GM advocates and therefore biases the results.
Answer Summary: Funding for the study was actually derived from a current supporter of GM technologies.
Detailed answer: It is clearly stated in the paper that the major funder of IHER?s involvement in the study is the Government of Western Australia, and the current governmentt is a supporter of GM crops.
With regard to IHER?s previous work in opposing Bt brinjal in India and CSIRO?s GM wheat in Australia, IHER conducted a thorough review of the evidence presented and concluded that there were serious safety concerns about GM brinjal and CSIRO?s GM wheat. The organization opposed the release of these based on a review of the evidence, not on ideology.
IHER is “a non-profit organization that has a scientific interest in the safety of genetically modified organisms (GMOs), particularly those destined for the food supply” according to Carman (http://gmojudycarman.org/
ML: The authors used ?statistical fishing? in their interpretation of the results, clearly attempting to skew or exaggerate their findings. What visual evidence is presented is done so to justify this statistical fishing experiment.
Summary: The authors executed careful and comprehensive statistical analysis to answer two hypotheses that had been generated by previous observations by the researchers in the U.S. piggeries.
Detailed answer: The authors performed statistical tests on all of the parameters that Mr. Lynas mentions, and none of them were found to be statistically significantly different. These analyses are clearly presented in the paper. Mr. Lynas either did not read the paper well enough or saw the analysis but did not understand them.
The counter argument from supporters of Mr. Lynas suggests that the study was not designed to test and statistically evaluate a sole hypothesis. If the authors had measured just the variables associated with the hypotheses being specifically tested (stomach inflammation and reproductive problems) and nothing else, few statistical tests would have been done and little to no statistical adjustment would have been suggested. The significant results that the authors found around the hypotheses that were tested should not be made invalid simply because the authors took some other measurements.
Furthermore, the level of inflammation in the non-GM fed group was concentrated in the mild to moderate range of inflammation. Feeding GM crops boosted that to severe inflammation, and this was a significant finding. Importantly, inflammation is a graded variable; the more inflammation, the more biologically impactful it can be to the animal. So, you cannot equalize the biological consequence of nil or mild inflammation to severe inflammation. Doing so goes against scientific knowledge on the effects of inflammation.
THIS is the definition of statistical fishing. There was a general hypothesis and they tested a lot of variables. You can not do that after the fact. Using standard statistical alpha values, if you measure 20 variables, at least 1 will be significant when it is really not. They should have had a hypothesis for each variable they measured or used a multivariate method that could account for the added variables, depending on the hypothesis.
As an example, I cite the relevant part from a statistics and experimental design book, “Experimental Design and Data Analysis for Biologists” by Gerry P. Quinn and Michael J. Keough (2002, ISBN 0521009766, p. 48):
“The Problem: One of the most difficult issues related to statistical hypothesis testing is the potential accumulation of decision errors under circumstances of multiple testing. As the number of tests increases, so does the probability of making at least one Type I error among the collection of tests.”
Feel free to consult any other statistics book.
TITLE: REPLY TO ANDREW KNISS? BLOG ON STATISTICS
SOURCE: GMO Judy Carman, USA
SUMMARY: “Andrew Kniss describes himself on his blog as a plant scientist. It is clear from this and other comments that he has made that he has little to no knowledge of medical research and the statistics used with it. But that doesn?t stop him proposing statistical methods that are not appropriate to this or any other data. [...] And while Andrew may not be aware of this statistical method, it is not hard for him to get information about it. It is even available on Wikipedia.”
REPLY TO ANDREW KNISS? BLOG ON STATISTICS
In his blog (weedcontrolfreaks.com), Andrew Kniss professes to know more about statistics than the authors of the paper, even though between them, two of the authors of the paper have 45 years experience in using and teaching statistics at a university level. They have expertise in agricultural, toxicological, medical and epidemiological statistics.
Everyone is human and many scientists make statistics errors. If they made an error doing a statistical fishing expedition, all the teaching experience is irrelevant. This is a difficult concept to grasp, as the book I cited above says. Why attack the person instead of the opinion?
When criticising uterine weight data, Andrew also says: ?Table 2 doesn?t list the units for any of the numbers, so I don?t know if the weights are in grams, kilograms, ounces, metric tons? ? Not only are some of his suggested units bizzare, he has made a fundamental mathematical and statistical mistake here. As is clearly shown in the caption to the table in question, the uterine weights are expressed as a percentage of body weight. Therefore the weights are expressed as a ratio and such data do not have units. Which is why no units were given.
“percentage of body weight” DOES have units, it is %, or in more detail, (weight of organ/weight of body). There are numbers that have no units, but it is because the algebra “cancels” the units, not because they are converted to something else. Carman and the others are the ones making the “fundamental mathematical mistake.”
Andrew Kniss describes himself on his blog as a plant scientist. It is clear from this and other comments that he has made that he has little to no knowledge of medical research and the statistics used with it. But that doesn?t stop him proposing statistical methods that are not appropriate to this or any other data. For example, he acknowledges that the stomach data are categorical in nature. But he then suggests using statistical tests that should never be used on categorical data, such as a t-test. In order to do that, he had tried to change categorical data into continuous data so that he can apply statistics that are only applicable to continuous data. Categorical data are data that fit into categories, such as male / female or pregnant / not pregnant. He has tried to turn this sort of data into data that is continuous, like you get with body weight or height. This is really bad statistical methodology. It is like taking pregnant / not pregnant data and trying to twist that data into groups that could be described as: pregnant, half pregnant and fully pregnant. And you are right, it doesn?t make sense to even try to do something like that.
A t-test is a t-test everywhere. Is there a publication that says this is different in medical studies?
And then, once he has ?succeeded? in turning the data into continuous data, the next step he should have done was to test the data to see if it is normally distributed or not, because different statistical tests are used on normally distributed data compared to data that are not normally distributed. But he hasn?t even done that. He just did both statistical tests ? one for normally distributed data and one for non-normally distributed data, and just reported both results.
Kinnis has provided a correction because a Wilcoxson test was more appropriate than his previously suggested t-test. But, still there was no statistical difference between the GM and the non-GM groups: http://weedcontrolfreaks.com/
Furthermore, the classification used for stomach inflammation, the main point of the paper, was divided in four categories, which is a weird way to separate it and it is not explained or justified in the methods. Kinnis shows that if the categories are just inflammation and non-inflammation, “the GM-fed pigs had LESS inflammation.”
In fact, the standard and appropriate statistical method that should be used with these sort of medical / toxicological data is the one that the authors used. It is called a ?relative risk? and it is calculated using a 2?2 table. This method takes into account the baseline risk of a condition or disease in the non-exposed group and calculates the extra risk of getting it in the exposed group. Andrew?s proposed method does not do that. And while Andrew may not be aware of this statistical method, it is not hard for him to get information about it. It is even available on Wikipedia.
Why reference Wikipedia (without providing a link to the particular article even) instead of a publication on statistics or methods?
Below are additional comments by me.
A question that Carman doesn’t answer: why that particular journal that is not even on Pubmed and most papers are pro-organics? The instructions of the journal say: “Topics are to be consistent with current principles of organic farming” (http://www.organic-systems.
If there is an effect of GM, this should be evident in a well done study. There are many questions around this particular paper which can explain why they found something that no one else has found, a possible negative effect of genetically modified products. If you have to massage the data so much, don’t understand experimental design and statistics, and send it to a journal that won’t question you too much (a technique made famous by the tobacco industry and used today by climate change deniers and creationists), then there is something fishy.
As for the claim that the paper was peer reviewed, peer review is not a magical recipe that cures all ailments in science. It is just that, peers that review the paper to find if things make sense. Previous biases, “pet theories”, personal grudges, and more, all enter into the process. After all, scientists are still human. This is why some people advocate for double-blinded and even for open peer review. For an example of a discussion on peer review, please check Nature’s special debate section: http://www.nature.com/nature/
Furthermore, peer review is not the last step, the paper has to survive criticism from more peers and other disciplines once it is published. For an example, see the “arsenic life” disaster: the paper was published in Science with the backing of NASA after being peer reviewed. In just a few days to a couple of weeks, it was completely debunked by a few of researchers on their blogs and, after a few more weeks, more formally in responses in Science itself. Many papers are criticized directly as responses (some journals have specific sections for this) and in other papers that discuss their relevance. Carl Zimmer has more info on the case of “arsenic life” at http://blogs.discovermagazine.
As an important point, Nature and Science are the top two scientific journals in the world. If peer review is a problem in those journals and they recognize it, it is a problem in all other journals too.
As a conclusion to this post, unfortunately the activist went off the rails in his response. First, questioning my intelligence and then implying that I was somehow part of the pro-GMO lobby, as if these were valid responses. These can only be the response of someone that doesn’t care about the evidence. How sad that so much energy is being wasted on fear and misinformation.
If we want to really protect the world and ourselves, science is the only proven way. Anti-GMO groups are anti-science when they lie about what studies say and defend a bad study just because it agrees with their ideas. As any other anti-science group, we have the duty to inform the public and challenge them to either prove any of their arguments or accept that they are not real.
Updated on 18-jun-2013 9:29am to fix a typo and some grammar.