Problems With the Peer Review System in Science

I have been reading a book by Mark Bowen, "Censoring Science", with the subtitle "Inside the political attack on Dr. James Hansen and the truth of global warming." I have to confess I have not read much of it, and that is partly because what I see in there is simply too disturbing to an innocent person such as myself, to read in big chunks. But while I have been randomly trying to make headway in the book, several related stories have caught my eye.

The first is a study in a peer reviewed Journal, with the hypothesis that remote prayer can help infertile couples conceive. According to the study, fertility rates are boosted 100%.

http://www.nytimes.com/2001/10/02/health/vital-signs-fertility-a-study-links-prayer-and-pregnancy.html
http://www.ebm-first.com/prayer/columbia-prayer-study-controversy.html

The second "The Truth Wears Off: Is there something wrong with the scientific method" by Jonah Lehrer, in the New Yorker Magazine, December 13, 2010. This article refers to something called "The Decline Effect". Which means that when a new scientific study is made it's effect decreases with time. To give a simple example (and this is made up, so don't look for it): If a scientific study on ESP finds that there is a person who can predict the next card off a deck 50% of the time, (compared to the average of 2%), then in future studies on the same subject, that figure of 50% will most likely decrease, and possibly drop down all the way down to 2%, thus eventually contradicting the results of the first study.

http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer?currentPage=1

It has been proposed that despite the efforts to eliminate bias in peer reviewed science, the bias still exists. But it drops with time, and not only that, but the bias reverses itself with time.

Why is that so? I think the Decline Effect is fairly easy to understand. Scientific studies that produce no result are rarely submitted to peer reviewed journals. For example, take the remote prayer study. If, say 100 independent studies had been made, and only one showed a positive effect, the one with the positive effect would show up in a Journal of Medicine, the other 99 would not even be submitted. That is the first level of bias. Then, if the study is kind of uninteresting, very few other scientists will attempt further tests on it. The peer review system does not compel other scientists to immediately duplicate any experiment to confirm or deny its hypothesis. So basically we have another level of bias, based on how interesting the experiment is to other scientists. Then, if an experiment leads to a general consensus, the bias swings the other way. That is, if remote prayer was accepted by the scientific community as more or less proven, more scientists would be encouraged to submit experiments proving that it was not true.

Many of these Decline Effects have to do with built in biases in the peer reviewed system. But I would imagine that the more important a study, or the more it makes sense, then the more scientists are attracted to it. And statistically, the more scientists that are interested, the better the peer reviewed system works. That is, you would have more studies submitted. Then you can better eliminate statistical outliers and mistakes. And with more research and debate, the better the understanding of the subject becomes.

We have some noticeable random glitches in the scientific peer reviewed process, when it comes to studies of low interest. I guess we can expect a "Decline Effect" with scientifically repulsive or uninteresting topics such as remote prayer. But with topics of extreme high interest such as global warming and evolution, the mass of research, and related experiments being done should help eliminate statistical aberrations in the system. Especially when those aberrations are directly related to the interest level.

Considering the general assault on science being conducted by the anti-global warming people and by Bible Literalists, all shoddy peer reviewed science is liable to be taken as a weakness of the entire structure of science. Any system, no matter how good, has weaknesses. And as it is exposed to attacks, those weaknesses will become more critical.

Recently we have seen leaked documents from global warming scientists, who seem to be expressing their frustration that science is being gradually undermined by outside forces. We are also starting to see pseudo-scientists finding ways to state that their experiments have been peer reviewed, even though they have not. No, it's not outright lying, it's simply twisting words around, half-truths, and looking for weak links in a complicated system.

For example, a person self-publishes a scientific experiment, then sends their book to an eminent scientist (albeit in another field) and asks for that scientist's opinion, then getting the opinion that "it looks interesting", calls it "peer reviewed". That is not peer review. But is it a lie?

The Peer Review system will probably need to set up a legal defence department one day to protect their good name against infringements. And scientists need to rigorously eliminate statistical anomalies like ESP experiments and Remote Prayer studies.

Just like Disney Corporation protects their Mickey Mouse character, and the Hell's Angels protect their death head logo, I think scientists need to protect their "Peer Review" brand. The world is changing, and science needs to keep up.

Another article about what's wrong with the peer review process and wondering how it can be improved.
http://whitehead.mit.edu/news/paradigm/spring_2008/peer_review.html

Picture: http://www.cartoonstock.com/directory/p/peer_review.asp