Confirmation bias

From a recent Amazon.com reader review of The Conscious Universe. This review provides a nice demonstration of the confirmation bias: If you believe that psi does not exist, but then you encounter evidence indicating that psi does exist, the confirmation bias will cause you to find reasons to reject the positive evidence, which in turn will make you more convinced -- and hence confirms -- that your original hypothesis was correct. Because this bias is rife throughout science, I discuss it and other biases in The Conscious Universe.

Incidentally, I doubt that the reviewer read the book carefully, because I never, ever use the term PSI. The correct word is psi. The reviewer writes (excerpt):

The main reason why I found this book unconvincing is that if PSI existed it would be easy to demonstrate. No meta-analysis of a huge number of studies is needed, one good experiment suffices.

This argument might seem appealing, except for one very important problem: Human performance is highly variable, even among those with world-class talent. E.g., some people claim to be able to hit baseballs thrown at them at 90 miles/hour. If that obviously ridiculous claim is true (because human reaction time is far too slow to hit such a fast-moving object), then it would be easy to demonstrate. We shouldn't need fancy baseball statistics. One good swing of a bat ought to prove the claim once and for all.

Back to the real world, we know that the best baseball players in the world get a hit only about once every three times at bat. If our skeptic happened to look at the plate only when the batter missed the ball, which was the majority of the time, he'd conclude that the baseball claim is false. Especially if he distrusted statistics.

The fact is that many excellent, highly significant experiments have been published, but "one good experiment" isn't convincing (and shouldn't be) because the results might have been due to an undiscovered flaw. This is why independent repeatability is important in science, but also why we always need a way to measure replication rates. Thus the need for meta-analysis. There's no way to escape this because all measurements contain error.

The test I found most impressive is also the simplest one: to have subjects mentally affect the result of random number generators. This experiment is easy and cheap to set up, is so simple that errors can easily be removed, and is easy to reproduce (the book claims that all subjects were successful in nudging the RNG outside of its operational parameters). If it works it would prove that PSI is real.

The book absolutely does not claim that all subjects are successful in these tests, nor does it claim that this is an easily reproducible effect. But the fact remains that some individual RNG experiments have provided extremely strong evidence in favor of psi, and yet as noted above skeptics dismiss those studies.

The fact that PSI believers instead of concentrating on one simple experiment are instead all over the place describing results of disparate, old, and complex studies is a bad sign.

This assumes that all researchers are clones and interested in exactly the same thing. Well, cancer is a very important health problem, but does that mean all biologists should be working exclusively on cancer research? Of course not. Researchers decide what they wish to focus on for dozens of reasons. Discomfort about "disparate, old, and complex" studies indicates a disinterest in history, avoidance of published experiments with strong results, and inability or unwillingness to understand experimental details that provide confidence that results are what they appear to be.

Another way to easily prove the existence of PSI would by using casino statistics. If there is one place where PSI should be present it is there. The author got some daily data about the money dropped and money won at different games such as roulette. If he had only gotten data about the number of roulette plays he could have easily demonstrated that the money won was more than expected thus demonstrating PSI. In the book he claims that casinos are reluctant to share such data, but I wonder why.

Do you know of any business that freely shares its detailed profit/loss details with anyone who asks? I don't. Casinos in particular are extremely tight-fisted about their data because they deal in cash. Public lottery information is another matter. For that data a few large studies have been conducted, as I describe in the book, and in fact they do provide evidence for psi.

So in conclusion, even this book did educate me about the arguments in favor of PSI and made me wonder and think about this issue, the net effect has been to strengthen my belief that PSI does not exist.

Lo! The confirmation bias at work.