Friday, December 03, 2010

My comments on Alcock's comments on Bem's precognition article

The Skeptical Inquirer (online version) has published a commentary on Daryl Bem's precognition research, written by dedicated skeptic James Alcock. The same story has also bubbled up to the attention of the mainstream media, including the New York Times and NPR. Some of the op-eds border on the hysterical. Others are more rational.

An example of the hysterical type includes Alcock's article. I will not address his critique of Bem's procedure (Bem does that calmly and effectively, hurting Alcock's feelings in the process), but I will comment on his preamble, which I reproduce here in indented blue text.
Parapsychology has long struggled unsuccessfully for acceptance in the halls of science. Could this article be the breakthrough? After all, it apparently provides evidence compelling enough to persuade the editors of that APA journal of its worthiness. However, this is hardly the first time that there has been media excitement about new "scientific" evidence of the paranormal. Over the past 80-odd years, this drama has played out a number of times, and each time, parapsychologists ultimately failed to persuade the scientific world that their phenomena actually exist. Recalling George Santayana’s now-clichéd dictum, "Those who cannot remember the past are condemned to repeat it,” we should approach Bem’s work with a historical framework to guide us. Consider the following:
DR: One reason for this failure are myths about psi research that are incessantly repeated by skeptics whose Bayesian priors are so close to zero that is it virtually impossible for any new evidence to sway their original beliefs. The strategy of repeating the same talking points ad nauseum is effective because most listeners eventually absorb those words as though they are true. A genuine skeptic would wonder if the critique offered by Alcock is backed up by solid facts. After doing some homework, he or she would eventually discover that most of it isn't.
  1. In 1934, Joseph Banks Rhine published Extra-sensory perception (Rhine & McDougall, 1934/2003), summarizing his careful efforts to bring parapsychology into the laboratory through application of modern psychological methodology and statistical analysis. Based on a long series of card-guessing experiments, he wrote: "It is independently established on the basis of this work alone that Extra-Sensory Perception is an actual and demonstrable occurrence.” (p. 210).

Elsewhere, he wrote: "We have then, for physical science, a challenging need for the discovery of the energy mode involved. Some type of energy is inferable and none is known to be acceptable….” (p.166)

Despite Rhine’s confidence that he had established the reality of extrasensory perception, he had not done so. Methodological problems eventually came to light, and as a result, parapsychologists no longer run card-guessing studies, and rarely even refer to Rhine’s work.

Is this conclusion sound? No. In Rhine's 1940 book, Extra-Sensory Perception After 60 Years, which refers to the period 1880 to 1940 (the authors were Pratt, Rhine, Smith and Stuart), Rhine et al discussed in great detail every critique their work had received and how potential design and analytical loopholes were addressed in subsequent experiments. They also listed all known replications of their card guessing method. This volume makes it clear that Rhine and his colleagues were as methodologically sophisticated and as hard nosed as their harshest critics, and that their data -- when viewed under the most critical light available at the time -- withstood those critiques.

One key reason why Rhine's work failed to sustain the initial excitement it had generated in the 1930s was the rise of behaviorism in academic psychology. Within that paradigm, not only was ESP considered to be flatly impossible, but any form of subjective experience, including conscious awareness, became forbidden topics. The reason few researchers today use ESP cards is not because the method was flawed, but because better methods were developed. Like any other area of research, methods and ideas naturally evolve and build upon the work of previous generations.

  1. Physicist Helmut Schmidt conducted numerous studies throughout the 1970s and 1980s that putatively demonstrated that humans (and animals) could paranormally influence and/or predict the output of random event generators. Some of his claims were truly extraordinary – for example, that a cat in a cold garden shed, heated only by a lamp controlled by a random event generator, was able through psychokinetic manipulation of the random event generator to turn the lamp on more often than would be expected by chance. His claim to have put psi on a solid scientific footing garnered considerable attention, and his published research reported very impressive p-values. In my own extensive review of his work (Alcock, 1988), I concluded that Schmidt had indeed accumulated impressive evidence that something other than chance was involved. However, I found serious methodological errors throughout his work that rendered his conclusions untenable, and the “something other than chance” was attributable to methodological flaws.

As with Rhine, excitement about Schmidt's research gradually dwindled to the point that his work became virtually irrelevant, even within parapsychology itself.

Accusations of "serious methodological errors" provide an easy justification for dismissing this remarkable body of work. But is it really true that such errors made Schmidt's work irrelevant, or that other researchers did not follow up his work? No. Hundreds of studies involving random number generators were published after Schmidt's studies, and meta-analyses of those experiments have been published and debated in mainstream physics and psychology journals. Schmidt's work inspired dozens of researchers to replicate and extend his work, and it continues to do so today in research programs like the Global Consciousness Project. His work has also spawned several psi-related patents.

  1. The 1970s gave rise to "remote viewing,” a procedure through which an individual seated in a laboratory could supposedly receive psychic impressions of a remote location being visited by someone else. Physicists Russell Targ and Harold Puthoff claimed that their series of remote viewing studies demonstrated the reality of psi. This attracted huge media attention, and their dramatic findings (Targ & Puthoff, 1974) were published in Nature, one of the world's top scientific journals. At first, their methodology seemed unassailable, but years later, when more detailed information became available, it became obvious that there were fundamental flaws in their procedure that could readily account for their sensational findings. When other researchers repeated their procedure with the flaws intact, significant results were obtained; with flaws removed, outcomes were not significant (Marks & Kamman, 1978; 1980).

Add Targ and Puthoff to the list of “breakthrough” researchers whose work is now all but forgotten.

Did supposed flaws adequately account for the results of remote viewing studies? No. Were those study designs abandoned? No. Did skeptics like Ray Hyman, who reviewed a small subset of the SRI/SAIC remote viewing studies for the CIA, conclude that the studies were flawed? No. Did this research paradigm, which was an updated version of picture-drawing techniques developed a half-century earlier, disappear? No.

Targ and Puthoff, and later Ed May and colleagues, continued not only to conduct substantial research on remote viewing, but it proved to be so useful for gathering information in a unique way that it was ultimately used for thousands of operational missions by the DoD. Some portions of the history of the formerly secret Stargate program (and other projects with different code names) is in the public domain now, so it is not necessary to go into that here. Suffice it to say that those research programs were very carefully monitored by skeptical scientific oversight committees who continued to recommend funding for over two decades (as long as the program remained secret).

  1. In 1979, Robert Jahn, then Dean of Engineering and Applied Science at Princeton University, established the Princeton Engineering Anomalies Research unit to study putative paranormal phenomena such as psychokinesis. Like Schmidt, he was particularly interested in the possibility that people can predict and/or influence purely random subatomic processes. Given his superb academic and scientific credentials, his claims of success drew particular attention within the scientific community. When his laboratory closed in 1970, Jahn concluded that: “Over the laboratory's 28-year history, thousands of such experiments, involving many millions of trials, were performed by several hundred operators. The observed effects were usually quite small, of the order of a few parts in ten thousand on average, but they compounded to highly significant statistical deviations from chance expectations. 3

However, parapsychologists themselves were amongst the most severe critics of his work, and their criticisms were in line with my own (Alcock, 1988). More important, several replication attempts have been unsuccessful (e.g., Jeffers, 2003, including a large-scale international effort led by Jahn himself (Jahn et al, 2000).

One more name for the failed-breakthrough list.

Other than the flub about a lab closing in 1970 that opened in 1979, again we see a cavalier dismissal of decades of research, implying that the work was systematically sloppy or methodologically naive or both. Nothing can be further from the truth. I was at Princeton for three years and spent enough time in the PEAR Lab to know that the research conducted there was as rigorously vetted and executed as any scientific project you will find anywhere. There weren't just "several replications." There were hundreds. The PEAR Lab's RNG research was a replication and extension of Helmut Schmidt's studies, and their remote perception research was a replication and extension of the SRI/SAIC remote viewing research. PEAR successfully and independently replicated both of those study designs. Even Jeffers, who Alcock cites to suggest that the PEAR RNG work could not be replicated, was later involved in a successful RNG experiment.

Do parapsychologists criticize each others' work? Of course they do. As in any scientific discipline, those who know the most are also the most qualified to provide critiques. This is healthy for advancing any field and for refining methods and interpretations, and such debates can be found in all areas of science and scholarship. It does not mean that colleagues are suggesting a wholesale dismissal of the evidence, as devout Skeptics are wont to do.

  1. In the1970s, the Ganzfeld, a concept borrowed from contemporaneous psychological research into the effects of sensory deprivation, was brought into parapsychological research. Parapsychologists reasoned that psi influences may be so subtle that they are normally drowned out by information carried through normal sensory channels. Perhaps if a participant were in a situation relatively free of normal stimulation, then extrasensory information would have a better opportunity to be recognized. The late Charles Honorton carried out a large number of Ganzfeld studies, and claimed that his meta-analysis of such work substantiated the reality of psi. Hyman (1985) carried out a parallel meta-analysis which contradicted that conclusion. Hyman and Honorton (1986) subsequently published a "Joint Communiqué" in which they agreed that the Ganzfeld results were not likely to be due to chance, but that replication involving more rigorous standards was essential before final conclusions could be drawn.

Daryl Bem subsequently published an overview of Ganzfeld research in the prestigious Psychological Bulletin (Bem & Honorton, 1994), claiming that the accumulated data were clear evidence of the reality of paranormal phenomena. That effort failed to convince, in part because a number of meta-analyses have been carried out since, with contradictory results (e.g., Bem, Palmer & Broughton, 2001, Milton & Wiseman, 1999). Recently, the issue was raised again in the pages of Psychological Bulletin, with papers from Storm et al (2010) and Hyman (2010). While the former argued that their meta-analyses demonstrate paranormal influences, Hyman pointed to serious shortcomings in their analysis, and reminded us that the Ganzfeld procedure has failed to yield data that are capable of being replicated by neutral scientists.

Because of the lack of clear and replicable evidence, the Ganzfeld procedure has not lived up to the promise of providing the long-sought breakthrough that would lead to acceptance by mainstream science.

Add Honorton (and Bem first-time-around) to the list.

Yes, Honorton and Hyman agreed that the results available in 1986 were not due to chance, and that confirmations with new data would be required to be persuasive. Honorton subsequently provided this successful prospective replication, and Bem and Honorton published it in 1993. That should be the end of the skeptical story.

But now we learn that the successful replication was in fact not persuasive because of publications that appeared six years later that presented "contradictory" results. Besides the retrocausal reason for dismissing a successful replication, is it really true that the meta-analyses were contradictory, or that avowed skeptics cannot successfully replicate the effect? No. Neither claim is true. The Milton & Wiseman (1999) analysis was flawed because it used unweighted statistics. When proper methods, based on a simple hit/miss count, are employed, that meta-analysis produces a statistically significant positive outcome. In fact, of the half-dozen meta-analyses of the ganzfeld database published to date, every single one is significantly positive (this is discussed in the online journal NeuroQuantology in an article by Tressoldi, Storm and Radin [Dec 2010, Vol 8 (4]). So rather than being contradictory, the existing ganzfeld database is actually completely consistent. In addition, skeptics have successfully repeated the ganzfeld experiment (it's not obvious from that article's abstract, but it is described in the paper itself).

What is the lesson from this history? It is that one should give pause when presented with new claims of impressive evidence for psi. Early excitement is often misleading, and as Ray Hyman has pointed out, it often takes up to 10 years before the shortcomings of a new approach in parapsychological research become evident.

In other words, Alcock suggests that we don't need to pay attention to new experimental data because it will probably, eventually, be shown to be flawed in some way. If promissory dismissals were regularly applied to any other area of science, everything would come to a grinding halt. No new findings would ever appear in any domain, because research methods are evolving and today's data and analyses are never going to be as good as tomorrow's.

One must also keep in mind that even the best statistical evidence cannot speak to the causes of observed statistical departures. Statistical deviations do not favour arbitrary pet hypotheses, and statistical evidence cited in support of psi could as easily support other hypotheses as well. For example, if one conducted a parapsychological experiment while praying for above-chance scoring, statistically significant outcomes could be taken as evidence for the power of prayer just as readily as for the existence of psi.

Yes, there can be many interpretations of experimental results. It is the investigators' job to devise methods that as clearly as possible distinguish between possible explanations. In any case, it is not necessary to have an explanation for observed results. If it were necessary then science would never had advanced. The moment theory is allowed to trump observations, science will collapse into a dogmatic religion.

Another key consideration: Parapsychology’s putative phenomena are all negatively defined – to claim that psi has been detected, all possible normal influences must be ruled out. However, one can never be certain that all normal influences have been eliminated; the reader of a research report has only the experimenter’s word for it.

This brings us to a related concern. Research reports involve an implicit social contract between experimenter and audience. The reader can only evaluate what has been put into print, and must presume that the researcher has followed the best practices of good research. We assume that the participants did actually participate and that they were not allowed to use their cellular telephones during the experiment, or to chat with other participants. We assume that they were effectively shielded from cues that might have inappropriately influenced their responses. We assume that the data were as reported, that none were thrown out because they did not suit the experimenter, and that they were analyzed appropriately and in the manner indicated. We assume that equipment functioned as described, and that precautions reported in the experimental procedure were carefully followed. We take for granted that the researcher set out to test particular hypotheses, and did not choose the hypotheses after looking at the data. We must take all this on faith, for otherwise, any research publication might simply be approached as a blend of fact, fantasy, skill, and error, possibly reflecting little other than the predilections of the researcher. Obvious methodological or analytical sloppiness indicates that the implicit social contract has been violated and that we can no longer have confidence that the researcher followed best practices and minimized personal bias. As Gardner (1977) wrote, when one finds that the chemist began with dirty test tubes, one can have no confidence in the chemist's findings, and one must wonder about other, as yet undetected, contamination. So, when considering the present research, we need not only to look at the data, but, following the metaphor, we need to assess whether Bem used clean test tubes.

This implies that it is possible to devise a perfect experiment. The clean test tube metaphor is a nice ideal, but in the real world there is no such thing. Nevertheless the ideal always gives the critic a convenient reason to dismiss any result that he or she prefers to disbelieve. If no actual flaw can be found, then one can always raise suspicions, or propose implausible scenarios that never actually occurred, or if all that fails, then they can just default to the catch-all criticism, "let's wait 10 years before we accept this result because by then someone will surely find something amiss."

In sum, Alcock has been an enthusiastic and effective defender of the skeptical faith for decades. His critiques are predictable and at first blush they may even seem reasonable, especially to those who aren't familiar with the research literature in question. But when you do know that literature, the arguments fall apart.

For a recent book that goes into the proponent/skeptic debate in some detail, including more of Alcock's critiques and my responses, I recommend this book:

Tuesday, November 30, 2010

Survival of the Godliest

Image source: here


I don't often cite articles, especially ones that are about religion, but this one I find fascinating. For those of us who pay attention to future possibilities, this could be a harbinger of a coming sea-change in the centuries-long tensions between science and religion. And in this scenario science doesn't win.

This blog article caught my eye because if we were living in a rational world one might imagine that support for the scientific study of unusual human experiences, including psi and mystical epiphanies, would be strongest among secular humanists, whose faith is in science (sort of). Unfortunately this is not the case, at least not among those who most vigorously wave the banner of secular humanism and proudly call themselves as "skeptics." They tend to view parapsychology as an illegitimate use of science to support religious beliefs. This is not true and never has been. But that's what they believe.

Religious people, by contrast, seem to split into two major groups. The ultra fundamentalists don't care about science at all, and they regard psi research as "testing" God. So they would like parapsychology, as well as many other things of the modern world, to just go away. More moderate theists are interested in seeing whether science can validate some of the tenets of their faith.

If the trend identified by the article continues, what will the world of the future look like? Mad Max? Logan's Run? The New Inquisition?

Monday, November 22, 2010

Wiseman - Bem exchange on "Feeling the Future"

Skeptical psychologist Richard Wiseman posted a critique of Daryl Bem's article, "Feeling the Future," which I've mentioned before on this blog. I won't repeat Wiseman's critique here, but the upshot of it is that in two of the experiments the investigator had the opportunity to correct subjects' misspellings of recalled words, and that, according to Wiseman, "This procedure presented an opportunity for subjective bias [by the experimenter] to enter the scoring system...."

This is a valid critique. Blind judging is preferred to avoid the possibility of such bias, and readers of the journal article would not have any way of judging whether the proposed biases actually occurred.

Bem provided this response to Wiseman (slightly edited by me for clarity):

This is a response to Richard (Wiseman’s) concern about the ability of the experimenter to correct misspelled words while being able to observe which corrections will help the psi hypothesis (because the misspelled word is a practice word) or work against the psi hypothesis. This is a legitimate concern and I will modify the database so that that the category information is not available to the experimenter when he or she makes spelling corrections.

The program that runs the experiment automatically calculates the results of the session, ignoring all words it doesn’t recognize as literal copies of the test words. This analysis is also transferred to the database, which is set up so that the experimenter cannot change it or any of the original words as typed by the participant. Any changes made by the experimenter in the database are explicitly shown as changes, and a security check flags records in which the experimenter has corrected any of the original words. In other words, there is a complete record of the original data that cannot be altered. As an additional check, the critical data appear in the output file in both unencrypted and encrypted form, and only I know the encryption formula. If anything is changed in the output, the security flag in the database will read “False.”

Any experimenter who wishes can simply ignore the option to correct misspellings. It will make little difference to the results, as the following shows.

My two experiments included 150 participants, who recalled a total of 2,920 words, of which 45 (1.5%) were misspelled. 23 of those were practice words and 22 of those words were non-practice control words, for a net “gain” of one word for the psi hypothesis. Here are the results reported in my article (in which I corrected misspelled words) compared with the original program-calculated results (which ignores all unrecognized words). The score is a Differential Recall% score, which can range from -100% to +100%, with scores > 0 being in the “psi-predicted” direction.

Experiment 8:
Corrected DR% score = 2.27%, t(99) = 1.91, p = .029, d = .19
Uncorrected DR% score = 2.29%, t(99) = 1.95, p = .027, d = .20

Stimulus Seekers: Corrected DR% = 6.46%, t(42) = 3.76, p = .0003, d = .57
Uncorrected DR% = 6.50%, t(42) = 3.91, p = .0002, d = .60

Experiment 9:
Corrected DR% = 4.21%, t(49) = 2.96, p = .002, d = .42
Uncorrected DR% = 4.05%, t(49) = 2.86, p = .003, d = .40

As can be seen, Experiment 8 is trivially hurt by the corrections; Experiment 9 is trivially helped.

Additional observations: Half of the words used in this experiment are common words, as determined by “Frequency Analysis of English Usage” by Francis and Kucera (e.g., apple, doctor) and half are uncommon (e.g., gorilla, rabbi) Although Richard uses "CTT" and "CAT" as examples to illustrate the ambiguity of correcting misspellings, in fact only a few different words were misspelled by anyone, and they are among the uncommon words or commonly misspelled words in the list (e.g., potatoe for potato). So, Richard’s hypothetical example, notwithstanding, in practice the correction of misspelled words is actually very straightforward and unambiguous. “Intrusions,” i.e. words that aren’t in the original list, are also very easy to spot. (I can furnish the list to whoever wants to try a blind correction exercise, but I don’t want to publish it here lest it ruin future participants.)

We see here one of the strengths of science in action. Sometimes a potential flaw turns out to be a show-stopper. Sometimes it doesn't. In this case, it doesn't.

Tuesday, November 16, 2010

Recent interviews


At this page you'll find a series of recorded interviews on the Essentials of Noetic Sciences, including episodes where I interview Daryl Bem about his new retrocausal experiments, Bruce Greyson on NDEs, and Rupert Sheldrake on his latest research.

Thursday, October 21, 2010

Extrasensory Perception and Quantum Models of Cognition

By Patrizio E. Tressoldi, Lance Storm, & Dean Radin.

The possibility that information can be acquired at a distance without the use of the ordinary senses, that is by “extrasensory perception” (ESP), is not easily accommodated by conventional neuroscientific assumptions or by traditional theories underlying our understanding of perception and cognition. The lack of theoretical support has marginalized the study of ESP, but experiments investigating these phenomena have been conducted since the mid‐19th century, and the empirical database has been slowly accumulating. Today, using modern experimental methods and meta‐analytical techniques, a persuasive case can be made that, neuroscience assumptions notwithstanding, ESP does exist. We justify this conclusion through discussion of one class of homogeneous experiments reported in 108 publications and conducted from 1974 through 2008 by laboratories around the world. Subsets of these data have been subjected to six meta‐analyses, and each shows significantly positive effects. The overall results now provide unambiguous evidence for an independently repeatable ESP effect. This indicates that traditional cognitive and neuroscience models, which are largely based on classical physical concepts, are incomplete. We speculate that more comprehensive models will require new principles based on a more comprehensive physics. The current candidate is quantum mechanics.

The full paper is available at the online journal NeuroQuantology.

Monday, October 18, 2010

Retrocausation in Psychology Today

This article in Psychology Today reports on Daryl Bem's new experiment on retrocausal effects (available on his website), in press in the Journal of Personality and Social Psychology. There are a few gaffs in the PT article, but overall it's quite positive.

Friday, October 15, 2010

Ganzfeld telepathy example 3

Another ganzfeld telepathy example with Gail and Tom. In this one Tom is the receiver.


Here is the transcript of what Ray said during the 10 minute sending period:

- - -
See a dark background. Colorful images. Very still. I feel outside. Air. Natural elements. Very, very clear, like sky. Something soft and football shaped. Orange-y, orange-y, light orange shape. I hear children. Maybe feathers, did I see feathers? Still see the dark background. Oval shape, oval shape. Images. Images on a background. Images on a dark background. Orange. Orange-y color. Air. Elements. Outdoors.

- - -

Can you guess the target that Gail was sending? When Tom viewed the four images he immediately selected the correct target with high confidence. What this shows is that the words that Tom chose to describe his mental impressions were accurate, but they didn't quite capture his actual experience. That experience allowed him to, as he put it, "bet money" that his selection was correct. And he would have won that bet.

Ganzfeld telepathy example 2

This ganzfeld telepathy test was conducted with Gail and her friend Ray, on September 27th.

This is the target pool. One of these images was being sent by Ray, from a distance. Neither Ray nor Gail knew anything in advance about the composition of the pool, and Ray selected one of these four pictures randomly with a tossed die.

What follows is the transcript of Gail's spoken impressions during the sending period, which lasted about 10 minutes:

- - -

I see an image of Ray when he jumped off the waterfall in Hawaii. Green. Swaying. I feel like I'm swaying. Palm Trees. Motion. I keep feeling like a lot of motion. Something solid and rectangular. Alive. Kinda...Jungle-y. Now my image changed to sort of like a port, but I feel like it's an analytical overlay. We were just talking about shipping so I'm not sure, all of a sudden I felt like I was at a port,
or a busy place or... Somewhere that felt like by, with water and industry. Hot.

Ray?! [At this point Gail felt that Ray's attention was wandering.]

Continuing with images of water or when I say like a sway or a wave, or some kinda flowing in and out feeling. I feel blues and whites. I'm not sure why I'm seeing an image of a stick of butter. But just that same shape again where I'm seeing a long, oblong... Before I felt like the oblong shape was standing upward and now I'm seeing it more like a train, you know, like a long train kind of a shape.

I'm hearing flower essence in my mind, flower essence. I see sort of a green leathery leaf. I guess I would draw a shape to go with that. A happy feeling.

Now all of a sudden, I just saw, I don't know, a metal structure, you know, like intricacies of a metal structure, that you would see like in an Eiffel Tower or a bridge or some sort of intertwining metal things.
Repeated design, or repeated. I'm not sure.

I just saw some horses galloping by, again, the same thing about swift. Something about just the mane blowing in the wind, or the hair. Something about the swift and swaying feeling of motion. I seem to be traveling from a tropical feeling to a sort of a farm feeling. Seems kinda busy. I still have to go back to my rectangular oblong shape. I don't know if stick-of-butter yellow has something to do with the color. Hands ... keep feeling like I'm holding something.

- - -

After the sending period, when Gail was trying to match her impressions to the target, she immediately rejected two of the pictures and kept vacillating between the actual target and one of the decoys. She first selected the target, then changed her mind, and went back and forth several times, ending up by selecting the decoy.

Can you guess which picture was the target that Ray was sending?

Monday, September 13, 2010

Ganzfeld telepathy example

Here's an example of data collected in a ganzfeld telepathy test session. This was conducted in our lab on September 10, 2010. My friend Gail was the receiver in this test; her friend Tom was the sender. This is Gail being prepped for exposure to the ganzfeld condition.



While Gail was going through a relaxation exercise, I led Tom to a distant location a floor above the lab and on the other side of the building. Once there, I gave him four opaque black envelopes, each of which had been prepared with a color photo inside, and then the four envelopes were thoroughly shuffled. Of course, neither Tom or Gail had any idea what those images might be. I gave Tom a die and asked him to toss it to get a random number 1 through 4. He did, and the photo inside that envelope became his target.

Tom then examined the photo and attempted to send it to Gail. Here is what Gail said during the 20 minute sending period, while under the Ganzfeld stimulation and listening to white noise played over headphones. Each sentence below is a continuous statement. The breaks between sentences indicate long pauses:

Keep feeling like looking up at tall, I'm looking up at something tall.


Something about texture. Texture.


I feel like something has a rough texture.


Tall, very tall impression, looking up high.


Feel as if I'm walking around observing something, like when you would walk in an art gallery or in a museum and you would look at something.


Wow.


First I'm feeling like tall trees, and then I'm feeling like tall building.


And then I'm like a Yosemite kind of image of a tall rock or a tall, some kind of a very tall solid stone something.


Seeing browns and grays


Something like a feeling of walking around, looking up and being in awe, in awe of something.


Monolithic or I don't know what the word is.


I'm getting images of Mount Rushmore, I know you're not supposed to say things [Gail was asked to avoid naming her impressions, as naming is known to often pull impressions into word-association fantasies].


Half-dome, like just a big stone.


I sort of feel like I'm walking around in a picture, and I'm giving my hand and we're climbing up ...


or something about going up, there's ....


It seems like there's also some kind of a round tall cylinder, and, something long and gray on the right.


Water fountain.


Stone.


At first I felt very much like I was in a nature, forest type of setting, ... and now I'm feeling more


something about a, like


a plaza


These are the four images in the target pool. Can you guess which one was the target?



Sunday, September 05, 2010

Told you so

"A series of quantum experiments shows that measurements performed in the future can influence the present." Read the article in Discover magazine.

Those of us conducting and publishing experiments in presentiment and precognition have been empirically demonstrating varieties of retrocausation for decades.

I look forward to the day when prejudices decline to the point that we don't have to wait for a few physicists to seriously entertain a topic before popular science editors feel comfortable enough to report on well established empirical effects.

Feeling the future

Daryl Bem's article, "Feeling the future," is now in press in the Journal of Personality and Social Psychology, an American Psychological Association high impact journal. You can download a preprint of the article from here.


Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect

Daryl J. Bem

Cornell University

The term psi denotes anomalous processes of information or energy transfer that are currently unexplained in terms of known physical or biological mechanisms. Two variants of psi are precognition (conscious cognitive awareness) and premonition (affective apprehension) of a future event that could not otherwise be anticipated through any known inferential process. Precognition and premonition are themselves special cases of a more general phenomenon: the anomalous retroactive influence of some future event on an individual’s current responses, whether those responses are conscious or nonconscious, cognitive or affective. This article reports 9 experiments, involving more than 1,000 participants, that test for retroactive influence by “timereversing” well-established psychological effects so that the individual’s responses are obtained before the putatively causal stimulus events occur. Data are presented for 4 time-reversed effects: precognitive approach to erotic stimuli and precognitive avoidance of negative stimuli; retroactive priming; retroactive habituation; and retroactive facilitation of recall. The mean effect size (d) in psi performance across all 9 experiments was .21, and all but one of them yielded statistically significant results. The individual-difference variable of stimulus seeking, a component of extraversion, was significantly correlated with psi performance in 5 of the experiments, with participants who scored above the midpoint on a scale of stimulus seeking achieving a mean effect size of .42. Skepticism about psi, issues of replication, and theories of psi
are also discussed.

Friday, August 27, 2010

Cover story


I'm the cover story for the North Bay Bohemian (the name seems appropriate for Northern California) newspaper. If you go here you can read the entire newspaper online, but I think this link will only last for one week.
Link

Monday, August 02, 2010

How to win a million dollars

Let's say we want to win a million dollar prize for rigorously demonstrating something psychic in a scientifically acceptable way.

One of the best candidates at present is the ganzfeld telepathy experiment. In this study two people are isolated, one is given the job of the "sender," the other the "receiver." The receiver is placed into a mild, unpatterned sensory stimulation condition called the ganzfeld, which produces a dream-like, hypnagogic state. In this state the receiver is asked to verbally report any impressions which come to mind. Meanwhile the sender is shown a randomly selected target image or video clip, and asked to mentally send that material to the receiver. After a half-hour of sending, the receiver is taken out of the ganzfeld and asked to select one of four images based on his or her impressions. One of those images is the target, along with three decoys.

By chance, the receiver will choose the actual target one in four times, or a 25% chance "hit rate." Because the result of each session can only be a direct hit or a miss, there is no ambiguity and the results are clean and straightforward to evaluate.

During the test session, the experimenter and receiver are of course blind to the sender's target image, and all parties involved are sequestered under rigorous conditions designed to preclude cheating. Within this design there is no way that receivers can systematically guess the correct target by conventional means.

A session typically takes about an hour for the two participants. For the investigator it takes another hour to prepare and to close down the session.

Analysis of over 4,000 such sessions, conducted in labs around the world for four decades, indicates that, with ordinary people involved as subjects, the observed average hit rate is 32%. This is stupendously significant from a statistical point of view, and it constitutes strong, repeatable evidence for telepathy.

How could we use this effect to win the million dollar prize?

First, we do a power analysis to determine how many repeated sessions we have to run. Let's say for a million dollars we are required to achieve results associated with odds against chance of a million to one. That seems like a reasonable criterion for success. Much less than this the prize holder runs the risk of losing a million to a fluke. Much more and the test becomes impossible to achieve during any one person's lifetime, as we'll see.

We'll design an experiment that is run in three phases, where each phase has the same parameters: p(chance) = 0.25, p(hypothesis) = 0.32, alpha = 0.003, power = 0.99. This means that if we assume that telepathy gives us a hit rate of 32%, then if we run this experiment we'll have a 99% chance of getting a final p-value of 0.003 or better, i.e. good evidence for telepathy. The power analysis tells us that we need to run N = 1,147 trials to achieve this result. So now we will run this same experiment two more times, get a result each time at least as good as p = 0.003, and then the combined p-value over all three phases will be one in a million or better, or odds against chance of at least a million to one.

This requires that we run a total of 1147 x 3 = 3441 sessions.

Say we pay each sender and receiver a modest $50 to help compensate their time and costs. So we need to budget $344,100 for participant compensation. And let's say we run one session per workday, and we pay our investigator $80 per hour. That comes to 688 weeks or 14 years of effort assuming we run the experiment 48 weeks per year. For the investigator (we'll assume one investigator, which is an underestimate), at two hours per session x 3441 sessions x $80, we end up with an investigator budget of $550,560.

Now we need a testing facility that provides exceptional security against cheating and will also allow independent observers, and perhaps the general public, to witness each session from afar. (Observers interested in monitoring this experiment are not going to camp out in the laboratory for 14 years to personally observe every session.) To do this, we could use a secure digital video recording system that streams encrypted data over the web to a secure site, and is also designed to detect any interception or tampering of the video record at the source, in transmission, or at the secure site. We will need at least three video streams to cover the sender, receiver, and investigator, and perhaps one or two more to survey the larger environment. So let's say we need five tamper-proof video streaming systems, a secure server, and expert consultants to ensure that the system is not only designed correctly, but is systematically re-examined to check for proper operation. Let's say all this costs $250,000 for the 14-year life of the project, which is undoubtedly a vast underestimate.

The testing facility itself should consist of two isolated chambers that are thoroughly shielded against any possible transmission of sensory cues, including sound, light, vibration, odor, magnetic or electromagnetic signals. This might cost $100,000. The cost of running the facility itself (air conditioning, electricity) might cost say $200 per week or $137,640 in total.

The total so far is $1,382,300, not including costs for independent analysis of the video, Internet bandwidth, storage costs, data analysis, etc. So, for the mere possibility of winning a million dollars, we're already deep in the red, not to mention the investment of 14 years of dedicated effort. In addition, to optimize the likelihood of success, we would want to recruit people who we think would perform well. Based on past studies this would be creative people (artists, musicians, etc.) who believe in telepathy because they've experienced telepathic episodes in their own lives, and/or siblings, spouses or friends who frequently experience telepathic-like connections. So we need an advertising and recruitment budget, and probably a travel budget as well to compensate selected participants who must travel to the testing facility.

In sum, based on the state of the science today, and based on decades of repeated experiments that give us some confidence on what to expect, I believe we could conduct an experiment that would win a million dollar prize (assuming we need odds against chance of a million to one). But from a pragmatic perspective it wouldn't be worth it. Even if it were possible to raise over a million to run the experiment in the first place (unlikely, unless a wealthy individual or foundation is interested in backing this project), this doesn't make either financial or scientific sense. It might generate some publicity for a day or two, but most people already believe in telepathy based on their personal experiences, so this would only be news to a small percentage of scientists who would raise an eyebrow for 10 minutes. And then they'd say "cool," and some of them would start working on explanatory theories. The hardcore skeptics, like flat-earthers and creationists, aren't convinced by evidence or prizes; they'll just go on believing whatever they want to believe.

If say, the X Prize Foundation, an organization with a good track record of offering clear challenges and criteria for success, were to establish a prize to demonstrate telepathy under rigorously controlled conditions, then I believe that the design I've sketched here would do the trick. For $10 million it would be worth it.

Saturday, July 31, 2010

Compassion for skeptics

I occasionally receive an email from someone who challenges me to apply for one or more of the so-called "prizes" offered by professional skeptics for demonstration of a psi effect. Here's an email exchange I had on this topic recently.

Mr X. wrote:
I saw your research in a new film last night …and was impressed with the rigor with which variables were limited, i.e. through a shielded room there was a clear transference of information between loved ones. Why not duplicate this experiment under the skeptical eye of James Randi and collect the million dollars being held by Goldman Sachs for just such a possibility?

http://www.randi.org/site/index.php/1m-challenge.html

If what you have created is repeatable and is as claimed in the film, you are the people to take this prize.

Please let me know your thoughts!
This sounded reasonable, so I replied:
Thanks for your feedback. The short answer to your question is that such prizes are effective for testing individuals who make claims that do not require skilled judging or statistical analyses. But they are not intended for serious scientific experiments. In addition, there is ample reason to believe that the "prize" is merely a publicity stunt. See this site for examples: http://www.skepticalinvestigations.org/exam/index.htm

From the above site: A leading Fellow of CSICOP, Ray Hyman, has pointed out, this "prize" cannot be taken seriously from a scientific point of view: "Scientists don't settle issues with a single test, so even if someone does win a big cash prize in a demonstration, this isn't going to convince anyone. Proof in science happens through replication, not through single experiments."
Mr. X's reply, using his text formatting and colors:

Thank you for your response, Dean. I have followed James Randi’s career since I was a teenager. I believe that he is a true student of the scientific method, as well as a careful writer, speaker and thinker.

“Ample reason,” as you write, is meaningless jargon. Your link takes me to a silly, paranormal proponent who calls himself Peabrain. There is no intellectual honesty in his writing or in your response.

“Such prizes,” as you say, points to an imaginary category that says nothing about Randi’s challenge.

I am always struck by how, as is exemplified in your film, science, as such, is defined as the materialistic thinking of limited imaginations and then, in the next scene, a purported scientific experiment, (redefining science for your purposes, more properly, as a method rather than a philosophy), is used to show the rigor of the logic and thinking that produces the results that titillate us, the audience, with evidence of paranormal phenomena. The woman who criticized science in general was the author of “The Field.” The pseudoscience was played up in the closing and latching of the big steel door.

Your results in this experiment are truly revolutionary. Bigger than big. Please repeat them so that they can be verified. Otherwise your entire organization is nonsense. And, the lady who puts the “correct information” into her healing drops? Are we supposed to nod our heads in agreement because we would otherwise undermine the placebo effect or is an intelligent adult human being really being asked to believe she can do just that? If the energy healer, Dr. Pearl, if I remember correctly, could actually do what he says he can do, it would be very easy to set up a scientific investigation of his claims and not rely on anecdotes . You belittle yourselves by aligning with such obvious charlatans.

Just demonstrate something! Prove something! Use independent observers we know and trust and show us something! It would be wonderful!! I am not a cynic! I am an open minded adult who wants to see demonstrations of the amazing nature of the universe that challenges everything I believe! What fun!! Are you just going to hide behind more hyperbole?

What if the challenge is intended for serious scientific experiments? What if the money is real? It would be easy enough for you to find out. The money can be verified through Goldman Sachs, says Mr. Randi.

Why not verify through, “replication, not through single experiments,” to borrow your own words, with independent verification of your results? It would be incredible. I would so love to believe your results, however, you give the thing person nothing to hang one’s hat on.

“A stunt,” you say? In the words of the one and only alien life form I know of, Alf, “Isn’t that the cat rancher calling the orbit guard a hairball?”

Respectfully challenging your claims,

I replied:
As I had noted, well known skeptic Ray Hyman said "Proof in science happens through replication, not through single experiments." I agree. And so several classes of independently conducted psi studies have been examined in detail to see if replication has been achieved. They have.

E.g., an article currently in press in Psychological Bulletin examines, among other things, the question of replication of one type of psi study (telepathy in the ganzfeld). It provides unambiguous evidence that this effect has been repeated over thousands of trials, in multiple laboratories around the world, and reported in over 100 publications for over 40 years. Another meta-analysis, including one published in the British Journal of Psychology a few years ago, indicates independent, significant replication of the effect of intention on a distant person's physiological condition. In addition, to date there have been six studies of various psi effects using functional MRI. Five of the six have shown significant evidence for psi. These analyses and studies are reported in scientific journals.

I could go on, but I provide detailed discussion of these and many other studies in my 2006 book Entangled Minds and 1997 book The Conscious Universe. Or you might look here (http://www.noetic.org/publications/journal_pubs.cfm) where you can find a few of our scientific and scholarly papers. If you want to hang your hat on scientific evidence, these resources are one place to start.

Other places to look: http://www.princeton.edu/~pear/publications.html. And many relevant articles can be downloaded from this site: http://www.scientificexploration.org/journal/articles.html

Mr X. then sends our correspondence (without my knowledge or permission) to Randi, and copies that note to me. He writes:
Hello Amazing,

I thought you’d enjoy this little repartee I had with the “scientists” at the Institute of Noetic Studies after viewing their recent, incredibly silly, movie, The Living Matrix



[Dean's note: This is a link to the movie. This movie was not made by the Institute of Noetic Sciences. It is one of many that some of our researchers appear in.]



P.S. As a lifelong rooter out of the ridiculous, I wonder if you are a regular lucid dreamer? I wonder if skeptics are more likely to question the fantastical events of their dreams than mystics.

Mr X. then adds to me:

I so much want someone to take this prize. To say it’s just a stunt and not demonstrate your field experiment a single time with the world watching and get Mr. Randi to pay or shut up is just a shame.

Here is a note from the man himself about our conversation:

Ray Hyman presumed that I had implied that a paranormal claim could be established or denied through one experiment. I had not.

However, the million-dollar prize IS awardable for one definitive success, as already clearly – and repeatedly – stated. Incidentally, it is no longer held by Goldman Sachs, but by Evercore Wealth Management, as Account #1007550. It currently stands at $1,137,082.32, though only one million is the actual prize amount. Details and certified documents are available on request.


I replied:

You would like to see a single demonstration win the prize. One experiment cannot establish anything, and thus cannot win the prize. Even Mr. Randi now admits this, as you show in your own email.

As I had noted below our work has been replicated many times and reported in peer-reviewed journals. If you choose to ignore that data and call what we do "silly," that's your prerogative, but it's neither science nor respectful.
Mr X. replied:

You are worse than silly and I am more than respectful. Randi admits nothing of the kind. He says “the million-dollar prize IS awardable for one definitive success,

(his emphasis). Why is your experiment demonstrable for your movie but not for independent verification?
And now Randi replies to Mr. X and copies to me as well:
I find it very significant that Dean Radin imposes his own rules, ignores the generous million-dollar prize, opts to believe that it doesn’t exist, and hides away hoping that it won’t be brought up again. Since he believes that movies prove matters, he must believe in Star Wars, as well…

They will continue to ignore the JREF prize. They have to, because they do not have the evidence that proves their case, and they know it. I wonder if they also believe that The Wizard of Oz is also a documentary? “Pay no attention to that man behind the curtain…!”

Finally, my last reply to Mr X, when it is clear that this is a waste of time:

You quote Randi as writing: "Ray Hyman presumed that I had implied that a paranormal claim could be established or denied through one experiment. I had not." (my emphasis)

Now you claim he says "
the million-dollar prize IS awardable for one definitive success."

Well, which is it?


And I'll leave it at that. This exchange demonstrates why I ignore "skeptics." Genuine skeptics don't speak this way. This is the language and style of scoffers and deniers. They hold on so tightly to their preferred view of the world that they can no longer respond rationally. They do not see that their approach is no different from extreme religious fundamentalists.

All I can do is offer them compassion.

Friday, July 09, 2010

Flash Mob Opera

This is not directly related to my interests in extended human capacities, but having been a concert violinist for many years I always appreciate new ways to popularize the experience of the classics (from La Traviata in this case to e.g., Sgt. Pepper’s Lonely Hearts Club Band). This video is one of the most delightful ways of popularizing classic opera that I've seen.


Wednesday, June 09, 2010

Maybe the check is in the mail?


The Wikipedia entry on Masaru Emoto is a good example of why no one should trust an encyclopedia written by anonymous amateurs. I know it is possible, at least in principle, to edit Wikipedia pages to make corrections. But it is also possible for pranksters to change information on any page just for fun. And I know teenagers who regularly do this to confuse their classmates.

The case in point was brought to my attention by a friend. I will correct the entry here. I've tried making corrections to Wikipedia in the past, and I'm not willing to go through that waste of time again. I'll italicize the Wikipedia entries:

In 2003, [the magician] James Randi publicly offered Emoto one million dollars if his results can be reproduced in a double-blind study.

I was coauthor on such a study, which was co-sponsored by the Institute of Noetic Sciences and published in 2006. You can find it here on PubMed. As far as I know Emoto hasn't received the one million dollar check. I know I haven't.

In 2006, Emoto published a paper together with Dean Radin and others in the peer-reviewed Explore: The Journal of Science and Healing (of which Radin was co-editor-in-chief).

Yes and no. Yes: I published such a paper. No: I became a co-editor-in-chief of this journal in 2009, three years after publishing that paper. I had no connection with Explore prior to that. Nor did I have any affiliation or financial interest in Emoto's work then, or now.

The paper itself was not peer-reviewed, as the journal only conducts peer reviews of articles submitted within the 'scientific' category, a label which Emoto and Radin chose not to apply to their work.


The citation attached to the above sentence refers to a photo essay about Emoto's crystals, published in Explore in 2004. I am not a coauthor of that article. I had nothing to do with it. The double-blind paper we published in 2006 was indeed peer-reviewed, and it showed a statistically significant difference between water that was "exposed" to intention vs. identical water set aside as a control. The magnitude of the observed effect was smaller than is implied in Emoto's books, but the direction of the effect was consistent with his claim.

A better-controlled "triple-blind" follow-up study published in the Journal of Scientific Exploration did not yield positive results.

No. The cited reference points to an article in a popular magazine that got it dead wrong. The abstract of the original journal article, of which I am a coauthor, reads:

An experiment tested the hypothesis that water exposed to distant intentions affects the aesthetic rating of ice crystals formed from that water. Over three days, 1,900 people in Austria and Germany focused their intentions towards water samples located inside an electromagnetically shielded room in California. Water samples located near the target water, but unknown to the people providing intentions, acted as ‘‘proximal’’ controls. Other samples located outside the shielded room acted as distant controls. Ice drops formed from samples of water in the different treatment conditions were photographed by a technician, each image was assessed for aesthetic beauty by over 2,500 independent judges, and the resulting data were analyzed, all by individuals blind with respect to the underlying treatment conditions. Results suggested that crystal images in the intentionally treated condition were rated as aesthetically more beautiful than proximal control crystals (p = 0.03, one-tailed). This outcome replicates the results of an earlier pilot test.

There were, however, potential problems with the "triple-blind" follow up. As the study explains:

Yes, as I explained. Empirical studies regularly contain sections discussing the limitations of the design. No experiment is perfect.

"In any experiment involving intention, the intentions of the "investigators" cannot be cleanly isolated from those of the nominal participants and this in turn constrains how one should properly interpret the results....

All quite true. But the above snippet starts in the middle of a paragraph. The actual article begins this paragraph with the following:

"These design elements excluded obvious environmental differences and conventional subjective biases as plausible explanations for the observed results, and the combined results of the two experiments appear to exclude chance as an explanation (unweighted Stouffer Z = 3.34, p<0.0004)."


In other words, when it comes to Wikipedia, reader beware. Meanwhile, I'm still waiting for my prize check. Maybe it's in the mail.

Thursday, April 29, 2010

Getting comfortable with stupidity

This is an excellent article describing why in science it is important to feel comfortable with one's stupidity (more like ignorance than stupidity). Non-scientists may not realize that most of the time in scientific research, especially research at the edge of the known -- which is where all the excitement is -- that we really don't know what we're doing. Those few things we think we do understand are taught in elementary college textbooks.

Students who do well in school, meaning those who get all the right answers on tests based on those textbooks, come to believe that they fully grok the nature of reality. But what they are grokking is what we thought we knew 10 or 20 years ago, and oftentimes textbooks are behind the curve of knowledge the moment they are published. Professors can't admit this, of course, because then students can argue that the tests aren't fair. So academia glosses over the fact that getting comfortable with stupidity is an extremely important lesson to learn, especially for smart people who intend to devote their lives to studying the unknown.

When smart people forget that ultimately we're all rather ignorant, that's when debates about controversial topics turn into emotional turf wars, and that's when real stupidity rears its ugly head.


Thanks to Damien Broderick for bringing this to my attention.


Martin A. Schwartz
The importance of stupidity in scientific research
Journal of Cell Science 121, 1771 (2008)
First published online May 20, 2008
Accepted 9 April 2008

I recently saw an old friend for the first time in many years.We had been Ph.D. students at the same time, both studying science, although in different areas. She later dropped out of graduate school, went to Harvard Law School and is now a senior lawyer for a major environmental organization. At some point, the conversation turned to why she had left graduate school. To my utter astonishment, she said it was because it made her feel stupid. After a couple of years of feeling stupid every day, she was ready to do something else.

I had thought of her as one of the brightest people I knew and her subsequent career supports that view. What she said bothered me. I kept thinking about it; sometime the next day, it hit me. Science makes me feel stupid too. It's just that I've gotten used to it. So used to it, in fact, that I actively seek out new opportunities to feel stupid. I wouldn't know what to do without that feeling. I even think it's supposed to be this way. Let me explain.

For almost all of us, one of the reasons that we liked science in high school and college is that we were good at it. That can't be the only reason - fascination with understanding the physical world and an emotional need to discover new things has to enter into it too. But high-school and college science means taking courses, and
doing well in courses means getting the right answers on tests. If you know those answers, you do well and get to feel smart.

A Ph.D., in which you have to do a research project, is a whole different thing. For me, it was a daunting task. How could I possibly frame the questions that would lead to significant discoveries; design and interpret an experiment so that the conclusions were absolutely convincing; foresee difficulties and see ways around them, or, failing that, solve them when they occurred? My Ph.D. project was somewhat interdisciplinary and, for a while, whenever I ran into a problem, I pestered the faculty in my department who were experts in the various disciplines that I
needed. I remember the day when Henry Taube (who won the Nobel Prize two years later) told me he didn't know how to solve the problem I was having in his area. I was a third-year graduate student and I figured that Taube knew about 1000 times more than I did (conservative estimate). If he didn't have the answer, nobody did.

That's when it hit me: nobody did. That's why it was a research problem. And being my research problem, it was up to me to solve. Once I faced that fact, I solved the problem in a couple of days. (It wasn't really very hard; I just had to try a few things.) The crucial lesson was that the scope of things I didn't know wasn't merely vast; it was, for all practical purposes, infinite. That realization, instead of being discouraging, was liberating. If our ignorance is infinite, the only possible course of action is to muddle through as best we can.

I'd like to suggest that our Ph.D. programs often do students a disservice in two ways. First, I don't think students are made to understand how hard it is to do research. And how very, very hard it is to do important research. It's a lot harder than taking even very demanding courses. What makes it difficult is that research is immersion in the unknown. We just don't know what we're doing. We can't be sure whether we're asking the right question or doing the right experiment until we get the answer or the result. Admittedly, science is made harder by competition for grants and space in top journals. But apart from all of that, doing significant research is intrinsically hard and changing departmental, institutional or national policies will not succeed in lessening its intrinsic difficulty.

Second, we don't do a good enough job of teaching our students how to be productively stupid - that is, if we don't feel stupid it means we're not really trying. I'm not talking about `relative stupidity', in which the other students in the class actually read the material, think about it and ace the exam, whereas you don't. I'm also not talking about bright people who might be working in areas that don't match their talents. Science involves confronting our `absolute stupidity'. That kind of stupidity is an existential fact, inherent in our efforts to push our way into the unknown. Preliminary and thesis exams have the right idea when the faculty committee pushes until the student starts getting the answers wrong or gives up and says, `I don't know'. The point of the exam isn't to see if the student gets all the answers right. If they do, it's the faculty who failed the exam. The point is to identify the student's weaknesses, partly to see where they need to invest some effort and partly to see whether the student's knowledge fails at asufficiently high level that they are ready to take on a research project.

Productive stupidity means being ignorant by choice. Focusing on important questions puts us in the awkward position of being ignorant. One of the beautiful things about science is that it allows us to bumble along, getting it wrong time after time, and feel perfectly fine as long as we learn something each time. No doubt, this can be difficult for students who are accustomed to getting the answers right. No doubt, reasonable levels of confidence and emotional resilience help, but I think scientific education might do more to ease what is a very big transition: from learning what other people once discovered to making your own discoveries. The more comfortable we become with being stupid, the deeper we will wade into the unknown and the more likely we are to make big discoveries.