Monday, November 22, 2010

Wiseman - Bem exchange on "Feeling the Future"

Skeptical psychologist Richard Wiseman posted a critique of Daryl Bem's article, "Feeling the Future," which I've mentioned before on this blog. I won't repeat Wiseman's critique here, but the upshot of it is that in two of the experiments the investigator had the opportunity to correct subjects' misspellings of recalled words, and that, according to Wiseman, "This procedure presented an opportunity for subjective bias [by the experimenter] to enter the scoring system...."

This is a valid critique. Blind judging is preferred to avoid the possibility of such bias, and readers of the journal article would not have any way of judging whether the proposed biases actually occurred.

Bem provided this response to Wiseman (slightly edited by me for clarity):

This is a response to Richard (Wiseman’s) concern about the ability of the experimenter to correct misspelled words while being able to observe which corrections will help the psi hypothesis (because the misspelled word is a practice word) or work against the psi hypothesis. This is a legitimate concern and I will modify the database so that that the category information is not available to the experimenter when he or she makes spelling corrections.

The program that runs the experiment automatically calculates the results of the session, ignoring all words it doesn’t recognize as literal copies of the test words. This analysis is also transferred to the database, which is set up so that the experimenter cannot change it or any of the original words as typed by the participant. Any changes made by the experimenter in the database are explicitly shown as changes, and a security check flags records in which the experimenter has corrected any of the original words. In other words, there is a complete record of the original data that cannot be altered. As an additional check, the critical data appear in the output file in both unencrypted and encrypted form, and only I know the encryption formula. If anything is changed in the output, the security flag in the database will read “False.”

Any experimenter who wishes can simply ignore the option to correct misspellings. It will make little difference to the results, as the following shows.

My two experiments included 150 participants, who recalled a total of 2,920 words, of which 45 (1.5%) were misspelled. 23 of those were practice words and 22 of those words were non-practice control words, for a net “gain” of one word for the psi hypothesis. Here are the results reported in my article (in which I corrected misspelled words) compared with the original program-calculated results (which ignores all unrecognized words). The score is a Differential Recall% score, which can range from -100% to +100%, with scores > 0 being in the “psi-predicted” direction.

Experiment 8:
Corrected DR% score = 2.27%, t(99) = 1.91, p = .029, d = .19
Uncorrected DR% score = 2.29%, t(99) = 1.95, p = .027, d = .20

Stimulus Seekers: Corrected DR% = 6.46%, t(42) = 3.76, p = .0003, d = .57
Uncorrected DR% = 6.50%, t(42) = 3.91, p = .0002, d = .60

Experiment 9:
Corrected DR% = 4.21%, t(49) = 2.96, p = .002, d = .42
Uncorrected DR% = 4.05%, t(49) = 2.86, p = .003, d = .40

As can be seen, Experiment 8 is trivially hurt by the corrections; Experiment 9 is trivially helped.

Additional observations: Half of the words used in this experiment are common words, as determined by “Frequency Analysis of English Usage” by Francis and Kucera (e.g., apple, doctor) and half are uncommon (e.g., gorilla, rabbi) Although Richard uses "CTT" and "CAT" as examples to illustrate the ambiguity of correcting misspellings, in fact only a few different words were misspelled by anyone, and they are among the uncommon words or commonly misspelled words in the list (e.g., potatoe for potato). So, Richard’s hypothetical example, notwithstanding, in practice the correction of misspelled words is actually very straightforward and unambiguous. “Intrusions,” i.e. words that aren’t in the original list, are also very easy to spot. (I can furnish the list to whoever wants to try a blind correction exercise, but I don’t want to publish it here lest it ruin future participants.)

We see here one of the strengths of science in action. Sometimes a potential flaw turns out to be a show-stopper. Sometimes it doesn't. In this case, it doesn't.

28 comments:

Vik said...

I'm new to statistical analysis and psi, but I came across this article relating to Dr. Bem's experiments.

http://www.ruudwetzels.com//articles/Wagenmakersetal_subm.pdf

I was wondering if you would take a gander and comment on it?

Dean Radin said...

I've already responded to that here: http://deanradin.blogspot.com/2010/09/feeling-future.html

Matt said...

Hi Dean,

A quick search didn't reveal any posts regarding this attempted replication:

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1699970

Do you have any thoughts about it?

Vik said...

Thanks, I read the thread, some good discussion. As an aside, I linked to that comment thread plus your talk on "Science and the taboo of psi" in a comment I posted in respose to this:

http://www.reddit.com/r/science/comments/ea3lt/remember_the_psychology_study_showing_that_humans/

Thank you.

Dean Radin said...

From the above paper:

> "Nevertheless, given that the history of Psi research is marked by subtle influences of experimenter bias ..."

This is a common belief that is false. Historical analysis of the use of blind procedures shows not only that these techniques were first developed for use in parapsychology, but it is actually used far more in parapsychology than in other contemporary fields. See e.g., http://tinyurl.com/2dlblwv

This experiment was conducted online, which is an important limitation as the authors noted. And yet they then claim "That we conducted a very close replication of Bem (2010, Study 8) and failed to obtain a reliable result."

The authors are cautious in their conclusions, and the study is valuable as a conceptual replication. But whether it is a "very close" replication is debatable.

Matt said...

"This experiment was conducted online, which is an important limitation as the authors noted"

Have you noticed reduced effect size for online experiments?

Dean Radin said...

Online performance-testing experiments are tricky to evaluate unless the subjects are in a controlled environment, because otherwise you never know how much attention they are placing on the task at hand.

Most people today multi-task when online, so if that was the case in this study, which requires focused attention and memorization, then the results could well go to chance. The authors of this paper were aware of this issue, and attempted to address it.

Sandy said...

One problem with Jeff Galak's online test is that you can cheat.

The online test shows you 48 words, which you are told to visualize. Then you are asked to write out as many as you remember. After the test is done, you are shown half of the words so that you can study them by practicing to write them out.

So this is a test of precognition to see if the words studied after the test are more likely to be remembered than the ones you did not get to study after the test.

The number of words remembered correctly (when you are asked to write down what you remember) is divided into two groups:
1) words you will be shown after the test is over
2) words you won't see again

If you remember more of the words you will be shown again after the test is over, you get a positive ESP score.

If you remember more of the words you are not shown again, you get a negative ESP score.

If the number of words in both groups is equal, the score is 0%. So if you have an excellent memory (or cheat by writing down the words as you are shown them) and can write out all 48 words, you will get a score of 0%. Conversely, if you don't remember any of the words (or just choose not to write any of them out), you also get a score of 0%.

You can’t cheat to increase the ESP score, because the list of words you get to study after the test changes each time. It is easy to cheat to get a null score because the list of 48 words is the same each time, and even if it wasn’t, you can still write down the words as you are shown them at the beginning of the test.

Cheaters can increase the likelihood of getting a null result from the experiment.

Dean Radin said...

Given Sandy's remarks, a skeptic of Wiseman's repute would surely rate that study as completely inadequate.

Mark Szlazak said...

Hello Dr. Radin and "Dr. Sandy."

This is a bit off topic but I was wondering if you could help me with articles or books on the "subtle (energy) body."

Specifically, the evidence for and against channels, meridians and vortex that are associate with "life force." A scholarly history of the subtle body would be nice as well.

I have some books but wondered which books or articles are considered best by the researchers here.

Thanks.

Dean Radin said...

The primary professional society interested in this topic is ISSSEEM: http://www.issseem.org/

I'm not an expert in subtle energies, but relevant books on my shelf include "Energy Medicine: The Scientific Basis", by James L. Oschman, "The Healing Promise of Qi: Creating Extraordinary Wellness Through Qigong and Tai Chi" by Roger Jahnke, and "Hands of Light: A Guide to Healing Through the Human Energy Field" by Barbara Brennan.

Sandy said...

Mark, I'm not a Dr yet. Nor am I a parapsychologist. I'm just a lowly PhD candidate who is currently on medical leave from my program. Hopefully, I'll sort things out and get back on track with my first love, mud. (Specifically, the sort of mud one collects from the bottom of lakes.) I like to think I have a talent for finding meaning in mud. :)

I think my knack for finding ways to cheat on tests comes from working as a TA throughout graduate school. I've been asked a number of times to evaluate on-line tests prior to them being given to university students. I'm good at finding problems with such tests before they are put into use.

I was going to recommend checking out ISSSEEM (http://www.issseem.org/), but I see Dean already did that. I seem to recall Bill Joines is doing some work in this area. The name William Tiller also comes to mind.

http://www.rhine.org/researchcurrent.htm

http://www.tillerfoundation.com/index.php

Mark Szlazak said...

Thanks Dean and Sandy for the pointers.

I like Oschman's book and look forward the the 2nd ed next year. Also, Brennan is good as well. I've not seen the "Healing Promise of Qi"

Tiller's books are really fundamental and I appreciate his work a lot.

Also, thank you for the lead to Bill Joines.

Another one I'm looking at is from Richard Gerber called "Vibrational Medicine." He has more on the association of channels found in various traditions with biological anatomy. It's more detailed which is what I like but I'm not sure if all these associations belong to the subtle body. Some maybe misinterpretations that really belong to the biological body.

Life force is considered as an essential element in psychic functioning by many traditions.
Dean, why don't you ever talk about life force in your parapsychology books?

Sandy, do you think Alex should talk about energy bodies on his show given all the NDE episodes he's had?

Sandy said...

Mark, I think Alex is more focused on basic survival of consciousness research with his Skeptico podcast. Energy bodies may be interesting, but I don't see his focus changing anytime soon.

Sandy said...

Mark, I forgot to mention that you could just ask Alex (and perhaps suggest a guest to be interviewed) if he would do an episode about research into energy bodies.

Alex is the expert when it comes to what will be on his show next, after all!

matthewx78 said...

I was just arguing with a guy in another forum about this, I basically told him that Wisemen is known pseudo-skeptic and he said I was making an Ad Hominem Fallacy.

Why should I bother when I know they constantly makes bogus Arguments!

The spelling corrects helped him out a wopping 1 point!!!!!!!!

http://www.evolver.net/user/matthew_x78/blog/daryl_bem_latest_mainsteam_replicate_esp_intview_about_his_work

matthewx78 said...

Mark Szlazak

In the 1960's a guy in China might have discovered "subtle channels" within the body. He needed to use special dyes to find these channels. He called the Bonghan (after his name) ducts after his name. There are tiny thread like structures inside of lymphatic vessels. Bonghan stated that these ducts correspond to all the acupuncture meridians. Recently researchers have been struggling to find these vessels over again and some have found them in rats.

Another theory is that there are "Organized Water Clusters" in the body which act as a communication medium in the body and here is a nice book on that.

http://www.amazon.com/Biophysics-Basis-Acupuncture-Health/dp/0974826103/ref=sr_1_1?ie=UTF8&s=books&qid=1290634103&sr=8-1

Personally my theory is that Acupuncture affects consciousness, or subtle energy directly through the nervous system, and possible Bonghan ducts or these organized water clusters. I think the best idea is that consciousness/subtle energy/morphogenetic fields surround and interpenetrate the body and acupuncture works as by affected the subtle energy.

John C. Eccles, a brain scientist, suggested that consciousness communicates with the nervous system and brain via the calcium or sodium ion, which is a quantum particle, or perhaps it does so in the synaptic space between axon and cell body of neurons. I postulate that energy work and or acupuncture affects the body in the very same way. In essence you might be boasting or tapering the field in that area.

My idea is speculative though.

Also check out Neuro Modular Technique. This is a technique where the acupuncturist or energy worker affects the patients nervous system with their intention.

Good Lick Mark. =]

Topher Cooper said...

If past history is any guide, the skeptical community will cite this as an unambiguous reutation of the experiment. Anybody knowledgable enough to cite demonstration that the refutation was refuted will either be ignored or dismissed as "post hoc fudging". If the point is pushed the "dirty test tube hypothesis" will be invoked: if any flaw is found, however trivial it is shown to be then it proves that the experiment was done slopily and therefore it can be assumed that there are unknown flaws that invalidate the experiment even though there are no significant flaws that anyone can find. If that is answered by pointing out that that is not how things are done elsewhere in science, "extraordinary claims requir extraordinary evidence" is invoked" (translation: "since the result can be assumed to be false any argument that is necessary to reach that conclusion is justified.")

Pikemann Urge said...

Mark, I suspect that Dean does not talk about life forces for either or all of these reasons:

1. It is beyond his competence
2. He has no current opinion
3. 'Life force' might be just a romantic term for psi
4. Psi is controversial enough - talking about a life force would make even more people dismiss Dean's work

Matthew, I hear you, and partly agree, but it does one better to stick to the data and the experiments and not so much the persons doing them. Wiseman has done some good stuff and he also holds his own opinions of psi too highly. I guess that's just being human!

Topher, you nailed it. An interesting point that I think I've stumbled upon: to hold a phenomenon as legitimate, you don't really need completely strict standards. E.g. measuring the speed of light; does light have a speed? But you do need consistency and independent verification. After that, strict standards are necessary to refine understanding.

And don't forget that hardcore skeptics will happily invoke the Dunning-Kruger effect whenever it suits them.

J.T. said...

The whole issue of PSI is difficult to measure in a laboratory basis on the extension of our 5 senses. However, the relationship between a newborn and the mother may be the area where a conclusive breakthough is made.
It has often been said that a mother will recognize the voice of her child and vice versa. I think the effect goes further and it would be interesting to note the biological effects of mother and child as a response to stimuli when neither can respond to the other in the standard 5 senses.
Has this ever been attempted?

Dean Radin said...

I'm not aware of any systematic psi studies that have involved the mother-newborn connection (at least in humans, there are rumors involving studies with rabbits but I don't know if they're true), but as it happens I was discussing some ideas about this very issue with colleagues yesterday. We may pursue this.

David Bailey said...

"I'm not aware of any systematic psi studies that have involved the mother-newborn connection"

Rupert Sheldrake has written on this, and I believe done some tests in relation to the milk let-down reflex.

Dean Radin said...

> Rupert Sheldrake has written on this ...

Yes. And he has a paper describing a survey of this phenomenon. But I'm not aware of any experimental tests.

matthewx78 said...

"extraordinary claims require extraordinary evidence"

Both Extraordinary claims and Extraordinary Evidence can become subjective terms..

If I tell you that there is a UFO outside by house this may sound extraordinary but if you went ahead and looked and went inside it yourself than you would known scientifically that it was true unless somehow it dreamt it up etc....

Pikemann -

Thanks for that, after reading this it seems that Wisemen had a decent crit. but it didn't turn out to alter results..

If I had more time I would have looked into myself when he said something.

I only share scientific info on PSI (and keep up with it) because it allows me to invite a new demographic to try to experience it form themselves. Most of the time I get put down but once in awhile...............................................................

I also rec. the IPOD ESP trainer/tester and this one

http://www.psychicscience.org/esp4.aspx

MickyD said...

http://www.guardian.co.uk/science/audio/2010/nov/29/science-weekly-podcast-memory

Chris French discusses Bem's work. Apparently he hasn't replicated the presentiment paradigm (not sure if this is true, as Louis Savva, one of his Ph.D students got sig results in one test). Oh well! The Bem stuff starts at 27:55.

Dean Radin said...

I just reviewed a paper reporting a successful independent replication of one of Bem's designs, but it hasn't been published yet so I can't go into details.

Enfant Terrible said...

Hi, Dean

could you tell us when will be published this sucessful replication? It will be published in Explore?

Best wishes.

Dean Radin said...

> could you tell us when will be published this sucessful replication? It will be published in Explore?

This wasn't submitted to Explore, although it would be welcome there. I don't know what journal the author has in mind, or when it will be published. Soon, I hope.