One of the most common arguments against a scientific finding is confirmation bias: the scientist or scientists only look for data that confirms a desired conclusion. Confirmation bias is remarkably common—it is used by psychics, mediums, mentalists, and homeopaths, just to name a few. As you may guess from such a list, deliberate use of confirmation bias is held in low esteem by scientists, and allowing confirmation bias to get the better of your results is regarded as a particularly sad form of incompetence.
Yet, whenever science meets some ideological barrier, scientists are accused of, at best, self-deception, and, at worst, deliberate fraud. Examples of this are scattered across the Internet with respect to evolution, gun control, sex education, and, of course, global warming. Let's take a look at three cases: in two cases, scientists were certainly duped by confirmation bias: the case of N-rays and homeopathy. In the last case—sex in Samoan society—we can see how difficult it can be to either establish or refute confirmation bias. I will then follow that up with a story from my own research, which shows how everyday scientific practice is designed to avoid falling into the trap of confirmation bias.
The amazing case of N-Rays
To understand how N-rays came about, we need to go back to the late 18th century and consider the cultural milieu of the time. The major European nations had their chests puffed out with pride: they were great and they knew it. Each was so great that they were convinced that they were greater than any of the others. National pride itself was even something to take pride in and polish up on Sundays like some classic car.
Scientists were a part of this, and national pride provided a significant motivation for their work. The UK was very happy with the likes of Faraday, Maxwell, and others, while the Germans had Hertz, Plank, and Roentgen, who had just discovered X-rays. The French may have felt a little left out in all of this, because, although they were making major contributions, they hadn't made as big of a splash as, for instance Roentgen and his X-rays.
That is, they hadn't until Prosper-René Blondlot announced the discovery of N-rays. He was immediately famous in France, and very shortly afterwards, researchers from around the world confirmed that they too had seen N-rays. N-rays were an ephemeral thing: observed only as a corona around an electric discharge from certain crystals. They were only observed by the human eye, making them difficult to quantify.
But not everyone was convinced. Many researchers outside of France were suspicious of the number of claims coming from French labs for the properties of N-rays. In the end, an American scientist Robert Wood visited the lab of Blondlot to see it for himself. During one of the experiments he surreptitiously removed the crystal that supposedly generated the N-rays, after which Blondlot failed to notice the absence of N-rays. The N-rays failed to vanish when their source was removed.
You might think that the story ends there, but it doesn't. National pride was such that some French researchers continued to publish research on N-rays in French journals for a number of years. If we look back, we can see how the N-ray fiasco developed—the French needed something stranger and more useful than X-rays. But we can also see why it collapsed—the experiments were readily repeatable, and there was a large, diverse, and active scientific community ready to put their oars in. In the end, only a small fraction of physicists studying in the area of radiation were taken in, and only for a short time.
Water memories
One of the most prominent examples of confirmation bias involved another French researcher, named Jacques Benveniste. He spent a great deal of time and effort studying the effect of histamines. Naturally, a histamine causes an anti-histamine reaction in certain tests. However, what Benveniste reported in Nature was that the reaction got stronger as the histamine solution was diluted—even when it was highly improbable that there was any histamine left in the solution. Water, in effect, had a memory of the histamine.
His research was published in spite of the fact that the reviewers expressed disbelief in the results, because they couldn't see an obvious flaw in the methods. Normally, the results would have been left to stand until independent researchers had either corroborated the finding or found the methodological flaw. In this case, however, Nature felt that the results had to be corroborated as quickly as possible. To achieve this, they sent a group of observers to examine the experiment in more detail.
They found that the positive result was due to inadequate blinding. The anti-histamine reaction had to be assessed by examining a reaction—in other words, there was a strong element of human judgement involved. And the researchers performing the analysis knew which samples should give a positive result. When the experiment was performed with complete blinding, the positive result disappeared.
The consequence of this was that the researchers involved in the Benveniste's work effectively withdrew from the scientific community. To this day, they still perform follow-up research on that original null result, making increasingly fantastical claims about homeopathic remedies. They, along with other homeopathic researchers, form a community apart. The community, as far as I can tell, has very little internal debate over findings, being neither critical nor receiving criticism.
The key point in these two stories is that confirmation bias was found rather quickly, and those scientists who refused to acknowledge it were quickly isolated from their peers. When controversial results turn up in good science, the result is rather different.
Free lovin' Samoans?
Our next example is of a woman named Margaret Mead who was an anthropologist. In the early 20th century, she landed in Samoa and interviewed a number of teenage girls on their view of sex and relationships. Being teenage girls, they, with her encouragement, may have spun her one hell of a line: Samoans appeared to be the ultimate free loving society. Girls started early, and got their kicks with anyone they pleased. Nothing was hidden, so there was no difficult transition through the adolescence years. Then, when the fun was over, the girls settled down to monogamous marriages.
Her book, first published in 1928, shocked and appalled conservative westerners. But it may have generated outrage in Samoa if the Samoans had known.
Modern Samoans are highly religious with strictly prescribed roles for boys, girls, men, and women. These roles certainly do not involve open casual sex—though statistics have shown that, just like all other societies, there is a fair percentage of adolescent promiscuity behind the scenes. In fact, when mixed with Western culture, the results were a bit dysfunctional, with some Samoan boys, in particular, coping poorly with a more openly promiscuous society.
This led to suspicion of Mead's results, and a New Zealand researcher named Derek Freeman returned to Samoa and re-interviewed some of the now elderly women that Mead had originally interviewed. In 1985, he published his own account detailing how Mead had got it wrong.
reader comments
211