In the philosophy of science, there have been many attempts to define what it is that distinguishes science from other attempts to know the world. There’s an interesting section of The Trouble With Physics where Lee Smolin discusses Paul Feyerabend’s work, and summarizes it something like this (I don’t have time to find the exact quote):

It can’t be that science has a method, because witch doctors have a method. It can’t be that science uses math, because astrologists use math. So what is it that distinguishes science?

Neither, so far as I know, came up with an answer. There is a hint in Smolin’s book that there is no answer; that each advance in science comes about because there is a weirdo whose approach to science works to make the discovery of the moment, but doesn’t work generally. This would explain why so few scientists tend to be really productive over their entire lives; usually they have a few productive years—maybe a productive decade or so—and then tend to fade: they spend a few years discovering everything that their personal quirks are suited to, then when it is exhausted, return to the normal state of discovering nothing.

There is something common, however, that one will find in all of these quirks, if one looks back over history. This is especially true if you go back far enough to notice how much of science turned out to be wrong. (It is a matter for another day that people take being wrong as one of the strengths of science, ignoring that a thing which may be wrong cannot be a logical authority, by definition.) There is one principle that you will find consistent between everything which has ever been science, right or wrong. That principle is: assume anything necessary in order to publish.

To see why, we must consider the evolutionary pressure that applies to science. For whatever reason, people rarely take the theory of evolution seriously. They consider it as a scientific doctrine, or an organizing principle for archaeology, or a creation myth or any number of other things, but very rarely as an operating force in the world. Yet selective pressures abound and have their effects.

Occasionally people will ask the question about what influence on science the academic doctrine of publish-or-perish has, and they are right to ask this, but it is really just a subset of a larger selective pressure: science consists exclusively of what is published. If someone were to do extensive research in his basement and discover all the secrets of the cosmos, but never tell anyone, none of his knowledge would be a part of Science. In the same sense that Chesterton said that government is force, Science is publication.

The big problem with trying to uncover the secrets of the cosmos is that they are well covered. Coming to know how the universe works is very difficult. It’s often much easier if one makes simplifying assumptions which get rid of variables or eliminate the need for expensive experiments because cheap ones will suffice. The problem is that an assumption being convenient is not a justification for making that assumption. But since science consists of what is published, there is a huge selective pressure on people to make these convenient assumptions. This may or may not influence any particular scientist, but the scientists who are willing to make these sorts of unjustified simplifying assumptions will certainly be included in Science, while the scientists who take the principled position and refuse to make unjustified assumptions may well not be, because they didn’t have results to publish. In fields where real results are difficult to come by, it’s entirely possible that this could come to dominate what is published. And as the pitchmen say, but wait, there’s more!

People who are willing to make unjustified assumptions tend to have some personality traits more than others. Arrogance and a certain sort of defensiveness tends to work well with making assumptions one can’t justify, since those discourage requests for justifications. It also works synergistically with making quick judgments based on superficial criteria (like holding unrelated unpopular opinions), since that tends to insulate the unjustified assumer from having to confront contrary arguments and evidence. And here we come to the question of evolution, because new scientists will have to get along with these people, since the scientists who have published largely serve as the gate-keepers of who gets to join science. What sort of candidates will these people accept? Who will find scientists like this tolerable?

In subsequent generations, there will be the further question of who will find tolerable the people who found the makers of unjustified assumptions tolerable? And so it will go through subsequent generations, each new generation being a mix of all sorts, but the presence of the makers of unjustified assumptions and those who they trained will act as as selective pressure even on those who don’t work with them directly, since they still must be able to work with these people as colleagues and in many cases submit journal articles to them for peer review, etc.

For any institution, if you want to know how it tends to go wrong, a good place to start is asking what are the selective pressures affecting it?