Tag Archives: Falsifiability

A meditation on physical units: Part 1

[Preface: A while back, Michael Raymer, a professor at the University of Oregon, drew my attention to a curious paper by Craig Holt, who tragically passed away in 2014 [1]. Michael wrote:
“Dear Jacques … I would be very interested in knowing your opinion of this paper,
since Craig was not a professional academic, and had little community in
which to promote the ideas. He was one of the most brilliant PhD students
in my graduate classes back in the 1970s, turned down an opportunity to
interview for a position with John Wheeler, worked in industry until age
50 when he retired in order to spend the rest of his time in self study.
In his paper he takes a Machian view, emphasizing the relational nature of
all physical quantities even in classical physics. I can’t vouch for the
technical correctness of all of his results, but I am sure they are

The paper makes for an interesting read because Holt, unencumbered by contemporary fashions, freely questions some standard assumptions about the meaning of `mass’ in physics. Probably because it was a work in progress, Craig’s paper is missing some of the niceties of a more polished academic work, like good referencing and a thoroughly researched introduction that places the work in context (the most notable omission is the lack of background material on dimensional analysis, which I will talk about in this post). Despite its rough edges, Craig’s paper led me down quite an interesting rabbit-hole, of which I hope to give you a glimpse. This post covers some background concepts; I’ll mention Craig’s contribution in a follow-up post. ]

Imagine you have just woken up after a very bad hangover. You retain your basic faculties, such as the ability to reason and speak, but you have forgotten everything about the world in which you live. Not just your name and address, but your whole life history, family and friends, and entire education are lost to the epic blackout. Using pure thought, you are nevertheless able to deduce some facts about the world, such as the fact that you were probably drinking Tequila last night.

The first thing you notice about the world around you is that it can be separated into objects distinct from yourself. These objects all possess properties: they have colour, weight, smell, texture. For instance, the leftover pizza is off-yellow, smells like sardines and sticks to your face (you run to the bathroom).

While bending over the toilet for an extended period of time, you notice that some properties can be easily measured, while others are more intangible. The toilet seems to be less white than the sink, and the sink less white than the curtains. But how much less? You cannot seem to put a number on it. On the other hand, you know from the ticking of the clock on the wall that you have spent 37 seconds thinking about it, which is exactly 14 seconds more than the time you spent thinking about calling a doctor.

You can measure exactly how much you weigh on the bathroom scale. You can also see how disheveled you look in the mirror. Unlike your weight, you have no idea how to quantify the amount of your disheveled-ness. You can say for sure that you are less disheveled than Johnny Depp after sleeping under a bridge, but beyond that, you can’t really put a number on it. Properties like time, weight and blood-alcohol content can be quantified, while other properties like squishiness, smelliness and dishevelled-ness are not easily converted into numbers.

You have rediscovered one of the first basic truths about the world: all that we know comes from our experience, and the objects of our experience can only be compared to other objects of experience. Some of those comparisons can be numerical, allowing us to say how much more or less of something one object has than another. These cases are the beginning of scientific inquiry: if you can put a number on it, then you can do science with it.

Rulers, stopwatches, compasses, bathroom scales — these are used as reference objects for measuring the `muchness’ of certain properties, namely, length, duration, angle, and weight. Looking in your wallet, you discover that you have exactly 5 dollars of cash, a receipt from a taxi for 30 dollars, and you are exactly 24 years old since yesterday night.

You reflect on the meaning of time. A year means the time it takes the Earth to go around the Sun, or approximately 365 and a quarter days. A day is the time it takes for the Earth to spin once on its axis. You remember your school teacher saying that all units of time are defined in terms of seconds, and one second is defined as 9192631770 oscillations of the light emitted by a Caesium atom. Why exactly 9192631770, you wonder? What if we just said 2 oscillations? A quick calculation shows that this would make you about 110 billion years old according to your new measure of time. Or what about switching to dog years, which are 7 per human year? That would make you 168 dog years old. You wouldn’t feel any different — you would just be having a lot more birthday parties. Given the events of last night, that seems like a bad idea.

You are twice as old as your cousin, and that is true in dog years, cat years, or clown years [2]. Similarly, you could measure your height in inches, centimeters, or stacked shot-glasses — but even though you might be 800 rice-crackers tall, you still won’t be able to reach the aspirin in the top shelf of the cupboard. Similarly, counting all your money in cents instead of dollars will make it a bigger number, but won’t actually make you richer. These are all examples of passive transformations of units, where you imagine measuring something using one set of units instead of another. Passive transformations change nothing in reality: they are all in your head. Changing the labels on objects clearly cannot change the physical relationships between them.

Things get interesting when we consider active transformations. If a passive transformation is like saying the length of your coffee table is 100 times larger when measured in cm than when measured in meters, then an active transformation would be if someone actually replaced your coffee table with a table 100 times bigger. Now, obviously you would notice the difference because the table wouldn’t fit in your apartment anymore. But imagine that someone, in addition to replacing the coffee table, also replaced your entire apartment and everything in it with scaled-up models 100 times the size. And imagine that you also grew to into a giant 100 times your original size while you were sleeping. Then when you woke up, as a giant inside a giant apartment with a giant coffee table, would you realise anything had changed? And if you made yourself a giant cup of coffee, would it make your giant hangover go away?

Or if you woke up as a giant bug?

We now come to one of the deepest principles of physics, called Bridgman’s Principle of absolute significance of relative magnitude, named for our old friend Percy Bridgman. The Principle says that only relative quantities can enter into the laws of physics. This means that, whatever experiments I do and whatever measurements I perform, I can only obtain information about the relative sizes of quantities: the length of the coffee table relative to my ruler, or the mass of the table relative to the mass of my body, etc. According to this principle, actively changing the absolute values of some quantity by the same proportion for all objects should not affect the outcomes of any experiments we could perform.

To get a feeling for what the principle means, imagine you are a primitive scientist. You notice that fruit hanging from trees tends to bob up and down in the wind, but the heavier fruits seems to bounce more slowly than the lighter fruits (for those readers who are physics students, I’m talking about a mass on a spring here). You decide to discover the law that relates the frequency of bobbing motion to the mass of the fruit. You fill a sack with some pebbles (carefully chosen to all have the same weight) and hang it from a tree branch. You can measure the mass of the sack by counting the number of pebbles in it, but you still need a way to measure the frequency of the bobbing. Nearby you hear the sound of water dripping from a leaf into a pond. You decide to measure the frequency by how many times the sack bobs up and down in between drips of water. Now you are ready to do your experiment.

You measure the bobbing frequency of the sack for many different masses, and record the results by drawing in the dirt with a stick. After analysing your data, you discover that the frequency f (in oscillations per water drop) is related to the mass m (in pebbles) by a simple formula:

where k stands for a particular number, say 16.8. But what does this number really mean?

Unbeknownst to you, a clever monkey was watching you from the bushes while you did the experiment. After you retire to your cave to sleep, the monkey comes out to play a trick on you. He carefully replaces each one of your pebbles with a heavier pebble of the same size and appearance, and makes sure that all of the heavier pebbles are the same weight as each other. He takes away the original pebbles and hides them. The next day, you repeat the experiment in exactly the same way, but now you discover that the constant k has changed from yesterday’s value of 16.8 to the new value of 11.2. Does this mean that the law of nature that governs the bobbing of things hanging from the tree has changed overnight? Or should you decide that the law is the same, but that the units that you used to measure frequency and mass have changed?

You decide to apply Bridgman’s Principle. The principle says that if (say) all the masses in the experiment were changed by the same proportion, then the laws of physics would not allow us to see any difference, provided we used the same measuring units. Since you do see a difference, Bridgman’s Principle says that it must be the units (and not the law itself) that has changed. `These must be different pebbles’ you say to yourself, and you mark them by scratching an X onto them. You go out looking for some other pebbles and eventually you find a new set of pebbles which give you the right value of 16.8 when you perform the experiment. `These must be the same kind of pebbles that I used in the original experiment’ you say to yourself, and you scratch an O on them so that you won’t lose them again. Ha! You have outsmarted the monkey.


Notice that as long as you use the right value for k — which depends on whether you measure the mass using X or O pebbles — then the abstract equation (1) remains true. In physics language, you are interpreting k as a dimensional constant, having the dimensions of  frequency times √mass. This means that if you use different units for measuring frequency or mass, the numerical value of k has to change in order to preserve the law. Notice also that the dimensions of k are chosen so that equation (1) has the same dimensions on each side of the equals sign. This is called a dimensionally homogeneous equation. Bridgman’s Principle can be rephrased as saying that all physical laws must be described by dimensionally homogeneous equations.

Bridgman’s Principle is useful because it allows us to start with a law expressed in particular units, in this case `oscillations per water-drop’ and `O-pebbles’, and then infer that the law holds for any units. Even though the numerical value of k changes when we change units, it remains the same in any fixed choice of units, so it represents a physical constant of nature.

The alternative is to insist that our units are the same as before (the pebbles look identical after all). That means that the change in k implies a change in the law itself, for instance, it implies that the same mass hanging from the tree today will bob up and down more slowly than it did yesterday. In our example, it turns out that Bridgman’s Principle leads us to the correct conclusion: that some tricky monkey must have switched our pebbles. But can the principle ever fail? What if physical laws really do change?

Suppose that after returning to your cave, the tricky monkey decides to have another go at fooling you. He climbs up the tree and whispers into its leaves: `Do you know why that primitive scientist is always hanging things from your branch? She is testing how strong you are! Make your branches as stiff and strong as you can tomorrow, and she will reward you with water from the pond’.

The next day, you perform the experiment a third time — being sure to use your `O-pebbles’ this time — and you discover again that the value of k seems to have changed. It now takes many more pebbles to achieve a given frequency than it did on the first day. Using Bridgman’s Principle, you again decide that something must be wrong with your measuring units. Maybe this time it is the dripping water that is wrong and needs to be adjusted, or maybe you have confidence in the regularity of the water drip and conclude that the `O-pebbles’ have somehow become too light. Perhaps, you conjecture, they were replaced by the tricky monkey again? So you throw them out and go searching for some heavier pebbles. You find some that give you the right value of k=16.8, and conclude that these are the real `O-pebbles’.

The difference is that this time, you were tricked! In fact the pebbles you threw out were the real `O-pebbles’. The change in k came from the background conditions of the experiment, namely the stiffness in the tree branches, which you did not consider as a physical variable. Hence, in a sense, the law that relates bobbing frequency to mass (for this tree) has indeed changed [3].

You thought that the change in the constant k was caused by using the wrong measuring units, but in fact it was due to a change in the physical constant k itself. This is an example of a scenario where a physical constant turns out not to be constant after all. If we simply assume Bridgman’s Principle to be true without carefully checking whether it is justified, then it is harder to discover situations in which the physical constants themselves are changing. So, Bridgman’s Principle can be thought of as the assumption that the values of physical constants (expressed in some fixed units) don’t change over time. If we are sure that the laws of physics are constant, then we can use the Principle to detect changes or inaccuracies in our measuring devices that define the physical units — i.e. we can leverage the laws of physics to improve the accuracy of our measuring devices.

We can’t always trust our measuring units, but the monkey also showed us that we can’t always trust the laws of physics. After all, scientific progress depends on occasionally throwing out old laws and replacing them with more accurate ones. In our example, a new law that includes the tree-branch stiffness as a variable would be the obvious next step.

One of the more artistic aspects of the scientific method is knowing when to trust your measuring devices, and when to trust the laws of physics [4]. Progress is made by `bootstrapping’ from one to the other: first we trust our units and use them to discover a physical law, and then we trust in the physical law and use it to define better units, and so on. It sounds like a circular process, but actually it represents the gradual refinement of knowledge, through increasingly smaller adjustments from different angles. Imagine trying to balance a scale by placing handfuls of sand on each side. At first you just dump about a handful on each side and see which is heavier. Then you add a smaller amount to the lighter side until it becomes heavier. Then you add an even smaller amount to the other side until it becomes heavier, and so on, until the scale is almost perfectly balanced. In a similar way, switching back and forth between physical laws and measurement units actually results in both the laws and measuring instruments becoming more accurate over time.


[1] It is a shame that Craig’s work remains incomplete, because I think physicists could benefit from a re-examination of the principles of dimensional analysis. Simplified dimensional arguments are sometimes invoked in the literature on quantum gravity without due consideration for their meaning.

[2] Clowns have several birthdays a week, but they aren’t allowed to get drunk at them, which kind of defeats the purpose if you ask me.

[3] If you are uncomfortable with treating the branch stiffness as part of the physical law, imagine instead that the strength of gravity actually becomes weaker overnight.

[4] This is related to a deep result in the philosophy of science called the Duhem-Quine Thesis.
Quoth Duhem: `If the predicted phenomenon is not produced, not only is the questioned proposition put into doubt, but also the whole theoretical scaffolding used by the physicist’.

Time-travel, decoherence, and satellites.

I recently returned to my roots, contributing to a new paper with Tim Ralph (who was my PhD advisor) on the very same topic that formed a major part of my PhD. Out of laziness, let me dig up the relevant information from an earlier post:

“The idea for my PhD thesis comes from a paper that I stumbled across as an undergraduate at the University of Melbourne. That paper, by Tim Ralph, Gerard Milburn and Tony Downes of the University of Queensland, proposed that Earth’s own gravitational field might be strong enough to cause quantum gravity effects in experiments done on satellites. In particular, the difference between the strength of gravity at ground-level and at the height of the orbiting satellite might be just enough to make the quantum particles on the satellite behave in a very funny non-linear way, never before seen at ground level. Why might this happen? This is where the story gets bizarre: the authors got their idea after looking at a theory of time-travel, proposed in 1991 by David Deutsch. According to Deutsch’s theory, if space and time were bent enough by gravity to create a closed loop in time (aka a time machine), then any quantum particle that travelled backwards in time ought to have a very peculiar non-linear behaviour. Tim Ralph and co-authors said: what if there was only a little bit of space-time curvature? Wouldn’t you still expect just a little bit of non-linear behaviour? And we can look for that in the curvature produced by the Earth, without even needing to build a time-machine!”

Artistic view of matter in quantum superposition on curved space-time. Image courtesy of Jonas Schmöle, Vienna Quantum Group.

In our recent paper in New Journal of Physics, for the special Focus on Gravitational Quantum Mechanics, Tim and I re-examined the `event formalism’ (the fancy name for the nonlinear model in question) and we derived some more practical numerical predictions and ironed out a couple of theoretical wrinkles, making it more presentable as an experimental proposal. Now that there is growing interest in quantum gravity phenomenology — that is, testable toy models of quantum gravity effects — Tim’s little theory has an excitingly real chance of being tested and proven either right or wrong. Either way, I’d be curious to know how it turns out! On one hand, if quantum entanglement survives the test, the experiment would stand as one of the first real confirmations of quantum field theory in curved space-time. On the other hand, if the entanglement is destroyed by Earth’s gravitational field, it would signify a serious problem with the standard theory and might even confirm our alternative model. That would be great too, but also somewhat disturbing, since non-linear effects are known to have strange and confusing properties, such as violating the fabled uncertainty principle of quantum mechanics.

You can see my video debut here, in which I give an overview of the paper, complete with hand-drawn sketches!


(Actually there is a funny story attached to the video abstract. The day I filmed the video for this, I had received a letter informing me that my application for renewal of my residence permit in Austria was not yet complete — but the permit itself had expired the previous day! As a result, during the filming I was half panicking at the thought of being deported from the country. In the end it turned out not to be a problem, but if I seem a little tense in the video, well, now you know why.)

Black holes, bananas, and falsifiability.

Previously I gave a poor man’s description of the concept of `falsifiability‘, which is a cornerstone of what most people consider to be good science. This is usually expressed in a handy catchphrase like `if it isn’t falsifiable, then it isn’t science’. For the layperson, this is a pretty good rule of thumb. A professional scientist or philosopher would be more inclined to wonder about the converse: suppose it is falsifiable, does that guarantee that it is science? Karl Popper, the man behind the idea, has been quoted as saying that basically yes, not only must a scientific theory be falsifiable, a falsifiable theory is also scientific [1]. However, critics have pointed out that it is possible to have theories that are not scientific and yet can still be falsified. A classic example is Astrology, which has been “thoroughly tested and refuted” [2], (although sadly this has not stopped many people from believing in it). Given that it is falsifiable (and falsified), it seems one must therefore either concede that Astrology was a scientific hypothesis which has since been disproved, or else concede that we need something more than just falsifiability to distinguish science from pseudo-science.

Things are even more subtle than that, because a falsifiable statement may appear more or less scientific depending on the context in which it is framed. Suppose that I have a theory which says that there is cheese inside the moon. We could test this theory, perhaps by launching an expensive space mission to drill the moon for cheese, but nobody would ever fund such a mission because the theory is clearly ludicrous. Why is it ludicrous? Because within our existing theoretical framework and our knowledge of planet formation, there is no role played by astronomical cheese. However, imagine that we lived in a world in which it was discovered that cheese was naturally occurring substance in space and indeed had a crucial role to play in the formation of planets. In some instances, the formations of moons might lead to them retaining their cheese substrate, hidden by layers of meteorite dust. Within this alternative historical framework, the hypothesis that there is cheese inside the moon is actually a perfectly reasonable scientific hypothesis.

Wallace and Gromit
Yes, but does it taste like Wensleydale?

The lesson here is that the demarcation problem between science and pseudoscience (not to mention non-science and un-science which are different concepts [2]) is not a simple one. In particular, we must be careful about how we use ideas like falsification to judge the scientific content of a theory. So what is the point of all this pontificating? Well, recently a prominent scientist and blogger Sean Carroll argued that the scientific idea of falsification needs to be “retired”. In particular, he argued that String Theory and theories with multiple universes have been unfairly branded as `unfalsifiable’ and thus not been given the recognition by scientists that they deserve. Naturally, this alarmed people, since it really sounded like Sean was saying `scientific theories don’t need to be falsifiable’.

In fact, if you read Sean’s article carefully, he argues that it is not so much the idea of falsifiability that needs to be retired, but the incorrect usage of the concept by scientists without sufficient philosophical education. In particular, he suggests that String Theory and multiverse theories are falsifiable in a useful sense, but that this fact is easily missed by people who do not understand the subtleties of falsifiability:

“In complicated situations, fortune-cookie-sized mottos like `theories should be falsifiable’ are no substitute for careful thinking about how science works.”

Well, one can hardly argue against that! Except that Sean has committed a couple of minor crimes in the presentation of his argument. First, while Sean’s actual argument (which almost seems to have been deliberately disguised for the sake of sensationalism) is reasonable, his apparent argument would lead most people to draw the conclusion that Sean thinks unfalsifiable theories can be scientific. Peter Woit, commenting on the related matter of Max Tegmark’s recent book, points out that this kind of talk from scientists can be fuel for crackpots and pseudoscientists who use it to appear more legitimate to laymen:

“If physicists like Tegmark succeed in publicizing and getting accepted as legitimate mainstream science their favorite completely empty, untestable `theory’, this threatens science in a very real way.”

Secondly, Sean claims that String Theory is at least in principle falsifiable, but if one takes the appropriate subtle view of falsifiability as he suggests, one must admit that `in principle’ falsifiability is rather a weak requirement. After all, the cheese-in-the-moon hypothesis is falsifiable in principle, as is the assertion that the world will end tomorrow. At best, Sean’s argument goes to show that we need other criterion than falsifiability to judge whether String Theory is scientific, but given the large number of free parameters in the theory, one wonders whether it won’t fall prey to something like the `David Deutsch principle‘, which says that a theory should not be too easy to modify retrospectively to fit the observed evidence.

While the core idea of falsifiability is here to stay, I agree with Scott Aaronson that remarkably little progress has been made since Popper on building upon this idea. For all their ability to criticise and deconstruct, the philosophers have not really been able to tell us what does make a theory scientific, if not merely falsifiability. Sean Carroll suggests considering whether a theory is `definite’, in that it makes clear statements about reality, and `empirical’ in that these statements can be plausibly linked to physical experiments. Perhaps the falsifiability of a claim should also be understood as relative to a prevailing paradigm (see Kuhn).

In certain extreme scenarios, one might also be able to make the case that the falsifiability of a statement is relative to the place of the scientists in the universe. For example, it is widely believed amongst physicists that no information can escape a black hole, except perhaps in a highly scrambled-up form, as radiated heat. But as one of my friends pointed out to me today, this seems to imply that certain statements about the interior of the black hole cannot ever be falsified by someone sitting outside the event horizon. Suppose we had a theory that there was a banana inside the black hole. To check the theory, we would likely need to send some kind of banana-probe (a monkey?) into the black hole and have it come out again — but that is impossible. The only way to falsify such a statement would be to enter the black hole ourselves, but then we would have no way of contacting our friends back home to tell them they were right or wrong about the banana. If every human being jumped into the black hole, the statement would indeed be falsifiable. But if exactly half of the population jumped in, is the statement falsifiable for them and not for anyone else? Could the falsifiability of a statement actually depend on one’s physical place in the universe? This would indeed be troubling, because it might mean there are statements about our universe that are in principle falsifiable by some hypothetical observer, but not by any of us humans. It becomes disturbingly similar to predictions about the afterlife – they can only be confirmed or falsified after death, and then you can’t return to tell anyone about it. Plus, if there is no afterlife, an atheist doesn’t even get to bask in the knowledge of being correct, because he is dead.

We might hope that statements about quasi-inaccessible regions of experience, like the insides of black holes or the contents of parallel universes, could still be falsified `indirectly’ in the same way that doing lab tests on ghosts might lend support to the idea of an afterlife (wouldn’t that be nice). But how indirect can our tests be before they become unscientific? These are the interesting questions to discuss! Perhaps physicists should try to add something more constructive to the debate instead of bickering over table-scraps left by philosophers.

[1] “A sentence (or a theory) is empirical-scientific if and only if it is falsifiable” Popper, Karl ([1989] 1994). “Falsifizierbarkeit, zwei Bedeutungen von”, pp. 82–86 in Helmut Seiffert and Gerard Radnitzky. (So there.)

[2] See the Stanford Encyclopedia of Awesomeness.

In search of the Scientific Method

`When I think of the formal scientific method an image sometimes comes to mind of an enormous juggernaut, a huge bulldozer — slow, tedious; lumbering, laborious, but invincible. […] There’s no fault isolation problem in motorcycle maintenance that can stand up to it. When you’ve hit a really tough one, tried everything, racked your brain and nothing works, and you know that this time Nature has really decided to be difficult, you say, “Okay, Nature, that’s the end of the nice guy,” and you crank up the formal scientific method.’    -Robert Pirsig

The first time I learned the Scientific Method was in high school. I was told to keep a logbook, in which I had to record the hypothesis to be tested, the apparatus used, the method of using said apparatus, the results, the discussion and finally the conclusion. It was incredibly boring. Also, in high school, it was pointless because you already knew what the outcome was going to be.

Unfortunately, this state of affairs remains basically true all the way through undergraduate studies in physics at University. Once again, it is tacitly understood that you are there to gain knowledge – pre-existing knowledge that can be found in a textbook — which was gained through the infallible Scientific Method. So it was quite a shock when I happened to take a History and Philosophy of Science course (purely optional for physics majors) and there learned that the Scientific Method simply did not exist.

Or rather, my mental image of the Scientific Method as a hard-and-fast list of rules, handed down through generations of scientists like the Hippocratic oath or the Ten Commandments, was a complete fiction. Instead, hordes of slavering philosophers clawed at each other, trying to define this mysterious procedure by which humans gained knowledge, that has come to be called `science’. Oddly enough, very few scientists seemed to be troubled by this, being too busy actually doing science to really worry about whether what they were doing was well-defined or not. In fact, the act of doing science comes so naturally to us that we frequently do not think to question how it is that we are able to make successful deductions about the world.

For example, suppose you notice that the rooster crows every morning just after the sun rises. You would probably deduce that the appearance of the sun caused the rooster to crow. However, suppose I told you that I had a big machine, and that there was a particular cog in the machine that would turn just before a bell rang. Since every cause should precede its effect, you could deduce that either the turning of the cog causes the bell to ring, or else there is some other component that is a common cause of both the cog turning and the bell ringing. Beyond that, we can say nothing about their relationship. So why do we not similarly think that there might be a common event that causes the sun to rise, and also makes the rooster crow?

The answer is probably that our brains have evolved to be naturally good at making deductions about the world, taking into account previous experience and the results of our interactions with the world. Our observations of the rooster and the sun take place in a larger context, in which we know quite a lot of stuff about the behavior of roosters and the sun relative to other things and we have built up a mental model of the world in which the rising sun triggers the rooster’s call. It is this very same mental model-building that we employ when we try to understand the natural world through science. We gather information, and then make deductions, partly using our existing intuitions and knowledge about the world, and partly using pure logic and statistics. The problem is thrown into particularly sharp relief when we try to build artificial intelligences (AI’s) that can do science and make deductions about the world. The trouble is that our AI’s do not have the benefit of millions of years of evolution built into them like we do, and so we have to tell them how to make sense of the world from scratch. If everything looked like cogs and levers to you, how would you make deductions about cause and effect? [1]

One of the most famous philosophers of science was Karl Popper. Popper argued that a key criterion of science is the fact that its hypotheses are falsifiable. In particular, whatever you might guess about the turning cog and the ringing bell, you should be able to do an experiment where you turn the cog and see whether or not the bell rings, and thereby eliminate one of your hypotheses. Unfortunately, this criterion is not good enough. For example, I can claim that there is a Bogeyman in my closet. This is clearly falsifiable – I just have to look inside my closet to determine whether or not the Bogeyman is present. However, it would not be correct to call this a scientific hypothesis, because there is absolutely no reason to think that there should be a Bogeyman there in the first place.

Thomas Kuhn took a different approach and tried to define science as a sort of social phenomenon with special characteristics. He argued that most science is more like puzzle-solving, where the goal is not to discover new rules by making hypotheses, but rather to resolve well-defined puzzles within an existing framework of rules that everybody agrees upon. In Kuhn’s paradigm, it is widely accepted that Bogeymen do not exist, so there is no Bogeyman puzzle to be solved.

Even physicists have got into the mix. David Deutsch has argued that we should prefer theories that are harder to alter in the face of new information. He points out that, having apparently falsified the Bogeyman theory, one could rescue it by claiming that the Bogeyman was invisible. This too could be falsified, if the poking of a pointy stick into the closet failed to elicit a response from the alleged Bogeyman, but it is clear that the vagueness in definition of the “Bogeyman” would always leave a possible way out for a theorist who did not want to accept the falsification. To avoid this, one should always prefer theories that are less amenable to variation. If I said instead that there was a giant diamond in my closet, then while it seems just as implausible as the Bogeyman Hypothesis, it is much more scientifically valid because a giant diamond has certain incontrovertible properties that cannot be amended in light of falsification (for example, diamonds are visible to the human eye, so if you don’t see it, it just ain’t there).

While there is no clear consensus on what exactly constitutes the scientific method, there are a few things that seem to be true about it. First, it is unlikely that one can characterize science by just a single criterion like Popper’s idea of falsifiability; a short list of characteristics is likely to do much better. Secondly, if you are not trying to fool anybody and you have a genuinely burning urge to discover the truth, and if in addition you are more or less rational and logical in your approach, then you will almost inevitably be following something like the scientific method. And finally, when in doubt, read detective stories. We all understand how Sherlock Holmes catches the bad guys and gets to the bottom of things: he gathers the facts and makes deductions, and whatever is left – “no matter how improbable” – is the truth. This process of information gathering and logical deduction that pervades detective fiction is also at the heart of the scientific method. And if you really want to see it laid out plain, you could hardly find a better reference than Robert Pirsig’s description, in Zen and the Art of Motorcycle Maintenance, of how a mechanic uses the scientific method to fix a motorcycle [2]. Here’s an excerpt:

“The real purpose of scientific method is to make sure Nature hasn’t misled you into thinking you know something you don’t actually know. There’s not a mechanic or scientist or technician alive who hasn’t suffered from that one so much that he’s not instinctively on his guard. That’s the main reason why so much scientific and mechanical information sounds so dull and so cautious. If you get careless or go romanticizing scientific information, giving it a flourish here and there, Nature will soon make a complete fool out of you. It does it often enough anyway even when you don’t give it opportunities. One must be extremely careful and rigidly logical in dealing with Nature: one logical slip and an entire scientific edifice comes tumbling down. One false deduction about the machine and you can get hung up indefinitely.”


[1] Michael Nielsen has a neat introduction to the AI community’s answer to this question.

[2] You can find the full excerpt here.