Antioxidant study in JAMA: flaws and subjective selection [ 28-03-2007 ]

The JAMA-study: a preliminary analysis

Drs. E. Valstar
In this systematic review and meta-analysis, published in JAMA (1) Bjelakovic et al analyzed a number of randomized studies concerning especially vitamin C, selenium, vitamin E, beta-carotene and vitamin A and they specifically looked at the death rate. Their analysis started with a number of selections however. Of the 815 trials 405 were excluded because no people died in these trials; 69 trials were not randomized; 24 delivered insufficient data (very subjective!); 4 were still ongoing and 245 did not fit their partly subjective criteria. According to the authors there were 68 trials of good quality left.

Multivariate analysis (an exact way to correct for a lot of bias and much more exact than univariate analyses) of these 68 trials showed no differences between antioxidants and placebo, with the exception of selenium: selenium significantly reduced the death rate in the 68 trials together. The question is of course: would this also have been found if the 245 studies mentioned would not have been excluded?
An objection against this kind of meta-analyses (also mentioned by Stampfer from Harvard) is: too many trials differ too much. If substance X in a small trial with people suffering from a special disease has an effect, while having no effect in a trial with a total different population, adding these trials will easily cover any significant effect of substance X. One can easily prove this with a chi-square-test (2). Therefore the positive overall result with selenium based on multivariate analysis is very remarkable.

Excluding 245 trials with a relevant death rate is ridiculous because a large number of trials –and especially large numbers of people– reduce standard errors considerably (see 2: standard errors can be divided by the root of the number of people in the concerning overall-group; e.g. if the total control group of 100 persons becomes 490,000, then any overall standard error can be divided by 70). Selection based on selective/subjective criteria undermines statistics.
Bjelakovic et al also performed a sub-subgroup analysis of their subgroup of 68 studies. Discussions between authors, while knowing the results of the studies, often determined in which sub-subgroup a trial was placed. This extra selection is even more ridiculous because Bjelakovic et al themselves perform two different statistic analyses (on page 845), not able to show heterogenicity among their 68 studies (this is an argument against their sub-subgroup analysis, although in my opinion such tests never can measure heterogenicity optimally). In their sub-subgroup analyses they also applied mostly univariate analysis and not (also) multivariate analysis. In the best trials, according to Bjelakovic et al (there is a lot of dispute about that), vitamin A and E had a disadvantageous effect on survival, but in other sub-subgroups it was the other way around. Furthermore I was not able to find any correction for the fact that more parameters, partly because of more groups, create a higher chance of coincidence-significance.

Bjelakovic et al should have restricted themselves to multivariate analysis of the 68 plus 245 is 313 studies. Bjelakovic et al were only chasing their own disbelief. This is confirmed by the fact that they could not find any relationship between death rate and duration of intake of the supplement(s). They even ignore more or less their most trustworthy result of selenium after multivariate analysis of the 68 studies and they selectively emphasize on univariate sub-subgroup analysis.

Why no meta-analysis with one parameter and in all trials the same main problem? With PSK this has already been done (3): with PSK the death rate of colorectal cancer, stage B/C is 15% lower. I can easily foresay: with melatonin for patients with metastasized cancer one will find prolongation of life. In my opinion it is not very likely that Bjelakovic et al will ever perform a meta-analysis of a substance that is likely to be favourable, because they do not belief in complementary medicine.

References
1. Bjelakovic G et al; JAMA 297:842-857, 2007.
2. Wijvekate ML: 'Verklarende statistiek'; Het Spectrum, Antwerpen, 1972.
3. Sakamoto J et al; Cancer Immunol. Immunother. 55:404-411, 2006.