In search of the Scientific Method

`When I think of the formal scientific method an image sometimes comes to mind of an enormous juggernaut, a huge bulldozer — slow, tedious; lumbering, laborious, but invincible. […] There’s no fault isolation problem in motorcycle maintenance that can stand up to it. When you’ve hit a really tough one, tried everything, racked your brain and nothing works, and you know that this time Nature has really decided to be difficult, you say, “Okay, Nature, that’s the end of the nice guy,” and you crank up the formal scientific method.’    -Robert Pirsig

The first time I learned the Scientific Method was in high school. I was told to keep a logbook, in which I had to record the hypothesis to be tested, the apparatus used, the method of using said apparatus, the results, the discussion and finally the conclusion. It was incredibly boring. Also, in high school, it was pointless because you already knew what the outcome was going to be.

Unfortunately, this state of affairs remains basically true all the way through undergraduate studies in physics at University. Once again, it is tacitly understood that you are there to gain knowledge – pre-existing knowledge that can be found in a textbook — which was gained through the infallible Scientific Method. So it was quite a shock when I happened to take a History and Philosophy of Science course (purely optional for physics majors) and there learned that the Scientific Method simply did not exist.

Or rather, my mental image of the Scientific Method as a hard-and-fast list of rules, handed down through generations of scientists like the Hippocratic oath or the Ten Commandments, was a complete fiction. Instead, hordes of slavering philosophers clawed at each other, trying to define this mysterious procedure by which humans gained knowledge, that has come to be called `science’. Oddly enough, very few scientists seemed to be troubled by this, being too busy actually doing science to really worry about whether what they were doing was well-defined or not. In fact, the act of doing science comes so naturally to us that we frequently do not think to question how it is that we are able to make successful deductions about the world.

For example, suppose you notice that the rooster crows every morning just after the sun rises. You would probably deduce that the appearance of the sun caused the rooster to crow. However, suppose I told you that I had a big machine, and that there was a particular cog in the machine that would turn just before a bell rang. Since every cause should precede its effect, you could deduce that either the turning of the cog causes the bell to ring, or else there is some other component that is a common cause of both the cog turning and the bell ringing. Beyond that, we can say nothing about their relationship. So why do we not similarly think that there might be a common event that causes the sun to rise, and also makes the rooster crow?

The answer is probably that our brains have evolved to be naturally good at making deductions about the world, taking into account previous experience and the results of our interactions with the world. Our observations of the rooster and the sun take place in a larger context, in which we know quite a lot of stuff about the behavior of roosters and the sun relative to other things and we have built up a mental model of the world in which the rising sun triggers the rooster’s call. It is this very same mental model-building that we employ when we try to understand the natural world through science. We gather information, and then make deductions, partly using our existing intuitions and knowledge about the world, and partly using pure logic and statistics. The problem is thrown into particularly sharp relief when we try to build artificial intelligences (AI’s) that can do science and make deductions about the world. The trouble is that our AI’s do not have the benefit of millions of years of evolution built into them like we do, and so we have to tell them how to make sense of the world from scratch. If everything looked like cogs and levers to you, how would you make deductions about cause and effect? [1]

One of the most famous philosophers of science was Karl Popper. Popper argued that a key criterion of science is the fact that its hypotheses are falsifiable. In particular, whatever you might guess about the turning cog and the ringing bell, you should be able to do an experiment where you turn the cog and see whether or not the bell rings, and thereby eliminate one of your hypotheses. Unfortunately, this criterion is not good enough. For example, I can claim that there is a Bogeyman in my closet. This is clearly falsifiable – I just have to look inside my closet to determine whether or not the Bogeyman is present. However, it would not be correct to call this a scientific hypothesis, because there is absolutely no reason to think that there should be a Bogeyman there in the first place.

Thomas Kuhn took a different approach and tried to define science as a sort of social phenomenon with special characteristics. He argued that most science is more like puzzle-solving, where the goal is not to discover new rules by making hypotheses, but rather to resolve well-defined puzzles within an existing framework of rules that everybody agrees upon. In Kuhn’s paradigm, it is widely accepted that Bogeymen do not exist, so there is no Bogeyman puzzle to be solved.

Even physicists have got into the mix. David Deutsch has argued that we should prefer theories that are harder to alter in the face of new information. He points out that, having apparently falsified the Bogeyman theory, one could rescue it by claiming that the Bogeyman was invisible. This too could be falsified, if the poking of a pointy stick into the closet failed to elicit a response from the alleged Bogeyman, but it is clear that the vagueness in definition of the “Bogeyman” would always leave a possible way out for a theorist who did not want to accept the falsification. To avoid this, one should always prefer theories that are less amenable to variation. If I said instead that there was a giant diamond in my closet, then while it seems just as implausible as the Bogeyman Hypothesis, it is much more scientifically valid because a giant diamond has certain incontrovertible properties that cannot be amended in light of falsification (for example, diamonds are visible to the human eye, so if you don’t see it, it just ain’t there).

While there is no clear consensus on what exactly constitutes the scientific method, there are a few things that seem to be true about it. First, it is unlikely that one can characterize science by just a single criterion like Popper’s idea of falsifiability; a short list of characteristics is likely to do much better. Secondly, if you are not trying to fool anybody and you have a genuinely burning urge to discover the truth, and if in addition you are more or less rational and logical in your approach, then you will almost inevitably be following something like the scientific method. And finally, when in doubt, read detective stories. We all understand how Sherlock Holmes catches the bad guys and gets to the bottom of things: he gathers the facts and makes deductions, and whatever is left – “no matter how improbable” – is the truth. This process of information gathering and logical deduction that pervades detective fiction is also at the heart of the scientific method. And if you really want to see it laid out plain, you could hardly find a better reference than Robert Pirsig’s description, in Zen and the Art of Motorcycle Maintenance, of how a mechanic uses the scientific method to fix a motorcycle [2]. Here’s an excerpt:

“The real purpose of scientific method is to make sure Nature hasn’t misled you into thinking you know something you don’t actually know. There’s not a mechanic or scientist or technician alive who hasn’t suffered from that one so much that he’s not instinctively on his guard. That’s the main reason why so much scientific and mechanical information sounds so dull and so cautious. If you get careless or go romanticizing scientific information, giving it a flourish here and there, Nature will soon make a complete fool out of you. It does it often enough anyway even when you don’t give it opportunities. One must be extremely careful and rigidly logical in dealing with Nature: one logical slip and an entire scientific edifice comes tumbling down. One false deduction about the machine and you can get hung up indefinitely.”


[1] Michael Nielsen has a neat introduction to the AI community’s answer to this question.

[2] You can find the full excerpt here.

4 thoughts on “In search of the Scientific Method

  1. Good stuff!

    You may be interested in another view of scientific revolutions…

    The scientific method has been described as gathering data. However, that understanding of the process is rather like saying, “Get lots of experimental data (then a miracle occurs) and out pops a useful theory.

    the missing piece of the puzzle is that there is also a way to understand the structure of theories and laws. Check out:

    There, i analyze the evolution of a theory of physics from ancient times through the scientific revolution. By focusing on causal relationships, and concatenated relationships between nodes, we gain rather useful insights into how to create more effective theories and policies.

    This is important because, within the social sciences, our current theories fail far more often than they succeed. imagine what we might be able to accomplish if our economic policies worked twice as well as they do? What about theories of management and psychology? Double the effectiveness and watch what happens to organizational and mental health!

    While the social sciences try very hard to follow the scientific method, they do not know how to make better theories. Instead, authors are happy to mix and mash theories and research – the ultimate goal is not to create a better theory – the goal is to get published, gain tenure, etc..



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s