I’ve had good results from using an extremely low-carb (i.e. low carbohydrate) diet to lose weight, so I’m highly skeptical whenever a study shows that they don’t do that. There are studies that show that they do, too, in addition to my experience, so something is going on when one has highly conflicting studies. The only thing to do is to actually dig into the studies.

And the thing one finds with many of the “low carb” diets in such studies is that they are frequently quite high carb. “Low carb” will often be defined as less than 100 grams of carbohydrate per day. People who have success eat well under 50 and frequently less than 20 grams of carbohydrate per day. A diet with 5-10 times as much carbohydrate being tested as a “low carb” diet simply doesn’t tell anyone anything useful.

But another big problem one sees is studies which test compliance at the same time they test efficacy. That is, the study breaks people up into groups and tells them what to do, but then records what they do as part of the group that they were assigned to. So if someone in the low carb group eats nothing but pasta, his weight performance will count to the low carb diet average in that study.

There are legitimate reasons for this, but they’re all for medical practitioners. Basically, such studies are useful to know how likely, if you prescribe a diet to all of one’s patients, is one to see results. Great for doctors, useless for the rest of us.

The other problem is that we largely already know what compliance with any behavioral change is in human beings: very low. It doesn’t much matter what you’re talking about, people don’t, typically, change for the better.

Where this is really egregious is where people look at these studies and don’t distinguish between the efficacy of the behavioral change and the degree to which the study told us what we already know about human beings: they don’t comply.

Hell, the compliance rates on taking a single pill a day are far from perfect; just look at all the people one knows who forget the pill from time to time. The compliance rate on 2x, 3x, and 4x pills per day is progressively worse, just from simple observation. Who, who has been proscribed taking a pill 3x per day, actually manages to take it 3x per day for all the days of the prescription?

When this comes to bigger stuff like diet and exercise, a simple and only somewhat inaccurate model is that people don’t comply. So a study which measures compliance + some change will mostly show no effect. But that’s uninteresting for people who will actually change.

Consider other areas of life: lifting weights or running. If you did a study to find out if lifting weights makes you stronger in which you also measured compliance, you’d find out that lifting weights doesn’t make you stronger. If you did a studying measuring whether running makes you a better runner, which also measures compliance, you’d find out that running doesn’t make you better at running. Hell, as long as the study is also measuring compliance, you’d find out that practicing piano doesn’t make you play piano better and taking dance lessons doesn’t teach you how to dance. Because in all these studies, the fact that most people stopped lifting weights, running, practicing piano, and never went to the dance lessons would dominate the results.

Or to put it simply, doing something only has an effect if you actually do it. No kidding.

Which is why what we need are studies which also measure compliance and separate people into groups based on compliance. This does introduce problems. Probably the biggest problem is that it will cost a lot of money because it will require really large groups of people. With 90%+ of people non-complying, you need a ten times larger group of people to study, and that costs a lot of money.

The second problem is that this switches out measuring the efficacy of the diet (or whatever) together with compliance for measuring the efficacy of the diet together with whatever preconditions (genetics, preferences, etc) make one likely to actually stick with it.

However, this is clearly a much more useful thing for an individual to measure. If I’m considering lifting weights, I want to know how much stronger I might get if I can stick with it. If I find I can’t stick with it, I don’t really care what it would do, anyway. And I don’t much care why I can stick with it, either.

If it turns out that only 10% of the population can stick with some diet, then I will consider taking my chances on finding out if I’m in the lucky 10%. Weight lifting works that way, to a limited degree. Everyone can get somewhat stronger, but only a fraction of the population can get hugely strong.

But there’s another issue at play, which has to do with motivation: knowing that something will work if I stick to it makes it vastly more likely that I will stick to it. If I actually believe that there is a causal connection between an action and a benefit, it is much easier to keep doing the action until I get the benefit.

Which is yet another reason that studies which measure compliance as well as an effect are worthless: the study participants didn’t know whether sticking to the plan even had any potential benefit.

So, in short, when it comes to studies showing no benefit to something, always check to see whether it’s a study that’s just telling you that human beings rarely change. It’s not completely worthless, but it’s only telling you what you already know.