I recently ran into an article on a study which compared the “Green Mediterranean Diet” with the Mediterranean diet and a “healthy diet”. The article begins

The green Mediterranean diet was pitted against the Mediterranean diet and a healthy diet in a large-scale clinical interventional trial- the DIRECT PLUS. Subsequent analysis found that the green Med diet reduced visceral fat by 14%, the Med diet by 7% and the healthy diet by 4.5%.

A bit later, we find out what the heck they mean by “green Mediterranean diet”:

The DIRECT-PLUS trial research team was the first to introduce the concept of the green-Mediterranean diet. This modified MED diet is further enriched with dietary polyphenols and lower in red/processed meat than the traditional healthy MED diet. On top of a daily intake of walnuts (28 grams), the participants consumed 3-4 cups of green tea/day and 100 grams (frozen cubes) of duckweed green shake/day. 

The first thing that jumps to mind is that the Mediterranean diet is popular among diet researchers because it is very low in red meat. How much lower could one make it? It is not plausible that a tiny amount of red meat causes enormous deleterious health effects since there’s obviously no corresponding dose-response to higher doses (people who eat a pound of red meat a day don’t die off at 15).

However that goes, worse is that this modification introduces three additional foods. This is the very opposite of controlling variables. Granted, the scientists in question probably think of it only as “introducing polyphenols”, but there’s a lot more in the foods they introduce than just polyphenols. Even worse, in terms of controlling variables, is that they are almost certainly not increasing the calories of the people on the diet, so they’re also going to be removing something or some things, introducing even more variables. You can tell that this is the case from the quote that they have from one of the professors who conducted the study (emphasis mine):

A healthy lifestyle is a strong basis for any weight loss program. We learned from the results of our experiment that the quality of food is no less important than the number of calories consumed and the goal today is to understand the mechanisms of various nutrients, for example, positive ones such as the polyphenols, and negative ones such as empty carbohydrates and processed red meat, on the pace of fat cell differentiation and their aggregation in the viscera

This strongly suggests that specific kinds of foods were removed from the diet at the same time that the polyphenols were added. That is really lousy variable control.

To be fair, it is possible to test many variables at once as a preliminary study to further, actually controlled studies, except that this isn’t a great way to do that unless you’re looking for something like an acute poison. If you design a study where on the one hand you test increasing protein intake and smoking cigarettes, and on the other hand reducing dietary trans-fats and moderate cocaine usage, who knows which group you’ll end up following up with, but either way you’re going to miss out on some important stuff.

Yet another variable in these kinds of studies is that they’re almost never free-feeding studies. By “free-feeding, I mean, “eat however much seems appropriate to you, of whatever food you’re hungry for/seems appropriate to you, whenever you’re hungry.” That is, free feeding is what normal people do. The number of people who actually weigh out all of their meals and eat according to some macro nutrient plan, every day of their lives, is approximately the number of bodybuilders there are. So we have yet another variable going on, which is that people who are watching their macro nutrient intake eat differently than people who don’t.

The reporting on this is also extremely lacking. For example, the summary at the top indicated a reduction in “red/processed meat”, while the professor who did the study referred to “processed red meat”. These are not the same things, and the top one is a much larger category. It includes fresh red meat and processed chicken, while the latter includes nothing that the former doesn’t too.

There’s also no mention of the limitations of the type of study performed. It’s an intervention study on 294 people (no indication if that was the starting or ending number) over 18 months. There’s no way that they had the money to keep the test subjects in a laboratory or hospital for all 18 months and strictly control all food given to them—not to mention, who would be willing to spend 18 months of their life under these conditions—so this had to have been an at-home study with self-reporting of compliance. Compliance rates for those are usually pretty bad, especially after the first month or two, and actual compliance as opposed to self-reported compliance is especially bad.

I’ve seen it argued that how difficult a diet (or other intervention) is to comply with is important, and this is certainly true. However, one of the things that affects compliance is the participant’s belief that compliance will reliably achieve something. Someone who wants to build muscle is far more likely to comply with a regimen of weight lifting because he is certain that if he complies, he will get the results he wants. An experimental method of muscle gain, where no one has any idea if it works, is far more psychologically difficult to comply with. Suppose one were to test out building muscle through twenty minutes a day of staring into the mirror and visualizing oneself with more muscle. Compliance with that will get difficult after the first day or two because there’s no obvious reason to keep going. (The flip side of this is that studies where one keeps participants in a lab and feeds them 100% of the food that they eat have phenomenal compliance, though higher drop-out rates, but they’re so expensive that in general they can only be run for 4-8 weeks and there’s very little of significance that one can find out in that short a span of time.)

The funding section is interesting, too:

This work was funded by grants from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project number 209933838- SFB 1052; the Rosetrees trust (grant A2623); Israel Ministry of Health grant 87472511; Israel Ministry of Science and Technology grant 3-13604; and the California Walnuts Commission.

None of the funding providers was involved in any stage of the design, conduct, or analysis of the study and they had no access to the study results before publication.

The California Walnuts Commission being one of the funders will, of course, sound alarms in the heads of the cynical or the conspiratorially minded. And, to be sure, the disclaimer that the funders of the study didn’t get to design it (etc) is no guarantee that the funders had no influence, because of course this isn’t the last study that the researchers are going to do and they will need funding again. By that same token, though, people who didn’t give a dime to this study may still have influenced the study by the researchers hoping to get funding from them next time. There is, as yet, no disclosure of funders who the researchers hope to get funding from in the future. All that said, it’s quite possible that the funding from the California Walnut Growers wasn’t that significant; they might well throw money at anyone who is trying to prove that walnuts are good for one’s health without overmuch worrying about the results of any one study. If you do fund enough studies looking into the health benefits of your product you’ll eventually get the favorable results you want by random chance, if nothing else, and this way your hands will be clean. It might even be cheaper, as it may potentially cost more to get someone with a decent reputation to falsify their results on purpose. But everyone makes mistakes.