Wednesday, March 24, 2010

More on the Harvard study on saturated versus polyunsaturated fats

This is a follow up on this post, which addressed the main argument put forth in a recent BBC article. The BBC article argued that people should replace saturated with polyunsaturated fats to reduce their risk of heart disease.

Let us take a look at the actual Harvard study itself (i.e., the study discussed in the BBC article). The Harvard study is linked here.

This post, by Stephan Guyenet, already pointed out several problems with the study. Stephan actually reviewed the studies used in the meta-analysis, and also some that were excluded in the meta-analysis and that he believes should have been included.

Here are a few other problems, in addition to the ones already pointed out by Stephan:

One thing that looks suspicious about this Harvard meta-analysis study is that they say that: “Statistical evidence for substantial between-study heterogeneity was not present (Q-statistic p = 0.13; I2 = 37%).”

A meta-analysis is a study that essentially summarizes, in a statistically sophisticated way, a bunch of other studies (the “sourced” studies). Too much between-study heterogeneity (i.e., widely disparate results among sourced studies) is undesirable, because it can bias the results.

The problem is similar to that of trying to summarize net worth figures (e.g., by calculating their average) in a middle class neighborhood that happens to have a few billionaires living in it. The heterogeneity in wealth may lead to a wildly overestimated average.

Now, we know that p values go down with sample size, and are usually high with small samples unless the effect measured by the statistic is very strong, regardless of the statistic used.

Well, with a sample of only 8 studies, their p value (associated with the Q statistic) is close to being significant at the 0.05 level!

If this sample of sourced studies were a little higher (say, 20), there would be significant between-study heterogeneity, which would call the meta-analysis into question. This is a big problem, since a good meta-analysis is expected to include a large number of studies (e.g., greater than 100), and this one included only 8 studies.

Moreover, to the best of my knowledge, the Q statistic is not very reliable when used with small samples, due to its low power as a test of heterogeneity. This makes the p value reported even more problematic.

Finally, the sourced study with the largest sample (n = 9,057; thus possibly the most credible), indicated as “Minnesota CS” on Figure 2 of the Harvard study, found increased risk of heart disease associated with increased consumption of polyunsaturated fats and reduced consumption of saturated fats.

Reference:

Mozaffarian, D., Micha, R., & Wallace, S. (2010). Effects on Coronary Heart Disease of Increasing Polyunsaturated Fat in Place of Saturated Fat: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. PLoS Med., 7(3): e1000252. doi: 10.1371/journal.pmed.1000252.

3 comments:

Anonymous said...

Thanks for that explanation Ned. I read your initial comments on Stephan's site, but your explanation here and the analogy used made it a lot clearer for a stats mong like me!

I spent a bit of time last night pouring through the search engine on the BBC site trying to find whether, for balance, 'the beeb' had ever reported on the Krauss meta-analysis from Jan 2010. No prizes for guessing the answer. Lots of 'saturated fat is going to kill you' type of reports, but nothing to balance that out.

Can't say I'm surprised however.

Ned Kock said...

Hi Jamie.

I still have a soft spot for the BBC. After all, these are the folks who gave us Monty Python!

On the other hand, it is no surprise that big media outlets are losing revenues. There are thousands of volunteer bloggers like me who can spend a couple of hours doing what the BBC writers should have done – try to understand the research they are talking about.

Anonymous said...

Argument is an intellectual process. Contradiction is just the automatic gainsaying of any statement the other person makes.

No it isn't.

Classic Python!

Thanks Ned!