One of the great things about science is that, when done properly, it’s easy to scrutinize it. So whenever you see someone cite a scientific study, always look into it. A friend recently gave me a link to this article in the NY Post titled, A Third of Women Only Date Men Because of the Free Food: Study. (note: he didn’t endorse it, just provided it for context).

If you look at the article, it links to this article in The Society for Personality and Social Psychology. This article describes the study in slightly more detail, but we need to look at the actual study, which is titled Foodie Calls: When Women Date Men for a Free Meal (Rather Than a Relationship).

So, first question: what was the study? (There were actually two, since my purpose is to illustrate why one should read the original paper critically, for brevity I’m going to only discuss the first study; go read the paper for the second one.) It was a survey of 820 women on Amazon’s Mechanical Turk service who were paid $.26 to answer a survey. (If you’re not familiar, Mechanical Turk is Amazon’s service where people are paid small amounts to do extremely short, simple tasks; it works because Amazon streamlines the process of getting many small tasks in succession so it’s worth it to the people doing it.) These were then filtered down to 698 self-identified heterosexual women. They were given personality questions as well as the question which makes the headline.

Have you ever agreed to date someone (who you were not interested in a relationship with) because he might pay for your meal?

Right off the bat, I dislike the phrasing on this because I’m used to “date” as a transitive verb meaning to be in a relationship with someone where the couple regularly go on dates. Which would make this question nonsense because it would be asking whether the women have been in a relationship with someone they were not interested in a relationship with. Clearly, by “date someone” they mean “go on a date with someone,” but this weird usage is going to influence how people respond. Among the possible reactions is to interpret the question more loosely, which means that both “yes” and “no” answers will mean a wider variety of things depending on how the responder interpreted the question.

And that’s apart from the way that people may well vary in interpreting the question. I could easily see women interpreting this to mean, “Did you ever go on a date with a man who hadn’t piqued your interest but, since he was paying for the meal, you thought you’d give him a chance to see if he improved on acquaintance?”

If what they wanted to ask was whether the woman ever intentionally misled a man into thinking she was open to a relationship with him when all she wanted as free food, why didn’t they ask that? Because such harsh language would color the results? Because if they said what they actually meant women might be embarrassed to admit it? So what was the goal? To try to trick them into revealing the truth?

I’m going to get back to that in a moment, but let’s take a short break to point out that when you read the paper, a third of women answered positively to the question, which only asks if they’ve ever done this even once. The study had a followup question about frequency; 20% of the women who went on a “foodie call” did so frequently or very frequently; since that’s 20% of 33%, that works out to 6.6% of all women. This is a long ways away from “a third of women only date men because of the free food.”

But back to the question: I imagine that people would try to defend the ambiguous language because words lie “deceive” imply judgement, and so will discourage respondents. Perhaps, but that’s because the thing being described is bad. Anyway way that the person understands of describing the intentional deception of a person to defraud them out of material goods will sound bad, because it is bad. The only way to make it sound not-bad is to phrase it in such a way that the respondent doesn’t know what you’re talking about.

Which gets me to the bigger point about this kind of psychological research: the simple expedient of phrasing your question ambiguously guarantees you publishable results. There’s no need to engage in p-hacking or other statistical tricks. Unlike with some of the stricter sciences like biology, getting fake results can be done with everything being completely above-board. It’s a great racket, which is why it will keep going for quite some time. Which is why you should never trust a summary of the results. Always track down the study and find out what the actual questions were.

Always question science. Good science is made to be questioned.