« Time, Consciousness and Moments of Truth | Main | Will Demise of Dollar Usher in Free Money? »


PrintPrinter-friendly version


Peer Review - Nature Initiates Trial and Debate

I have long been suspicious of peer review and its darker side - suppression of valid but non-conforming points of view in science. In an article published on my health site I included a proposal by David Kaplan first published in The Scientist, on what to do to 'fix peer review'. Kaplan suggests that the two functions of peer review - improving manuscripts and judging their scientific merit - often interfere with each other and should be separated.


scribe.jpg

The Scribe - Image credit: Eric Hotz


Evidently, Kaplan is not the only scientist calling for a review of peer review. Nature now has, perhaps somewhat reluctantly, scheduled a trial of "open peer review", where

"authors can choose to have their submissions posted on a preprint server for open comments, in parallel with the conventional peer review process. Anyone in the field may then post comments, provided they are prepared to identify themselves. Once the usual confidential peer review process is complete, the public 'open peer review' process will be closed. Nature will report on the results after the trial period is over."

The 'debate' page asks questions such as

What is the best method of peer review?

Is it truly a value-adding process?

What are the ethical concerns?

And how can new technology be used to improve traditional models?

- - -

Here is how Philip Campbell PhD, Editor-in-Chief of Nature, describes the experiment:


Nature's peer review trial

Peer review is the bedrock of scientific publication (for Nature's position on peer review, see our Guide to Authors). It is widely considered essential for improving submitted papers and enhancing the credentials of scientists as well as those of the journals in which they choose to publish.

But, like any process, peer review requires occasional scrutiny and assessment. Has the Internet brought new opportunities for journals to manage peer review more imaginatively or by different means? Are there any systematic flaws in the process? Should the process be transparent or confidential? Is the journal even necessary, or could scientists manage the peer review process themselves?

Nature's peer review process has been maintained, unchanged, for decades. We, the editors, believe that the process functions well, by and large. But, in the spirit of being open to considering alternative approaches, we are taking two initiatives: a web debate and a trial of a particular type of open peer review.

The trial will not displace Nature's traditional confidential peer review process, but will complement it. From 5 June 2006, authors may opt to have their submitted manuscripts posted publicly for comment.

Any scientist may then post comments, provided they identify themselves. Once the usual confidential peer review process is complete, the public 'open peer review' process will be closed. Editors will then read all comments on the manuscript and invite authors to respond. At the end of the process, as part of the trial, editors will assess the value of the public comments.

At the close of the trial, we will assess the value of public comments overall as well as the practicalities of their inclusion on a longer-term basis. We will publish an account of the trial and our conclusions.

For further details about the trial for authors and reviewers, please go to the question and answer page provided. Further general questions not answered by this list can be directed to the editors. Questions from authors and referees about particular manuscript submissions should be sent to the editor who is handling the manuscript.

(... on Nature.com)


Update December 2006:

Science journal 'Nature' abandons peer review experiment
Citing a lack of participation, the British journal Nature said it was ditching a closely watched online experiment that allowed scientists to comment on their peers' research before publication. The four-month trial, which began in June, was aimed at democratizing the peer review process, a time-honored tradition in which a group of select scholars critique scientific manuscripts and decide what appears in print.

- - -

Although Nature ended its experiment, the end of peer review as we know it seems to be steadily approaching. Here is an interesting article that appeared in USA Today - January 2007:

Is this the end of the scholarly journal?

By Gregory M. Lamb, The Christian Science Monitor

Scientific advances sometimes come as lightning flashes of inspiration. But when scientists sit down to record and take credit for what they've found, they still use much the same method they have for decades - an article published in a scholarly journal.

But science's hidebound traditions are changing. The Internet has opened up new forms of publishing in which anyone in the world can find and read a scientific paper. And papers themselves are becoming more interactive, leading readers to the underlying data, videos, and discussions that augment their value. With blogs and e-books providing easy means of self-publishing, some observers are speculating that scholarly journals and their controversial system of peer reviews may not be needed at all.

"The traditional journal publishing medium we've grown used to really needs to evolve and change because that's not the way people are accessing information," says Mark Gerstein, a professor of biomedical informatics at Yale University in New Haven, Conn. Dr. Gerstein cowrote an article, "The Death of the Scientific Paper," which appeared last year on The-Scientist.com, an online science magazine.

If the hopes of innovators bear fruit, scientific advances will come ever more quickly as online publishing makes past research easier to access and share widely.

Two new scientific publications, both available only online, may signal what's ahead. The PLoS ONE (plosone.org), a journal begun by the Public Library of Science (PLoS) last month, aims to put as many new scientific articles as possible on the Internet to be read by anyone, free of charge. The Journal of Visualized Experiments, or JoVE (myjove.com), is a kind of YouTube for researchers. It operates on the theory that a short video showing how an experiment is done is better than thousands of words that attempt to describe it.

At PLoS ONE, which aspires to be a general science journal along the lines of Science and Nature, the papers themselves are only a starting point. Readers can annotate, comment on, and critique the findings: Their contributions become permanently attached to the original article. At least one commentator has likened this process to a kind of "electronic Talmud," in which the original document receives elaborate commentary and discussion that over time adds greatly to its value.

In coming months, says Chris Surridge, the managing editor of PLoS ONE, readers also will be able to rate papers on their quality, such as how surprising or groundbreaking the results were - much in the way Netflix subscribers rate movies they rent using one- to five-star ratings. In this sense, PLoS ONE is moving toward a Web 2.0 model, which focuses on user-generated content strategies already used by websites such as Digg.com, Slashdot.org, or Amazon.com.

For years, traditional "peer review" has come under fire. A jury of three experts, the peer reviewers, assess each article and recommend only those that they feel represent the most significant new work. At many elite scientific journals, fewer than 10% of the articles submitted are accepted. Many of the rejected articles eventually travel down the "food chain" to be published in a plethora of less prestigious (and less noticed) specialty journals.

A year ago, the respected U.S. journal Science was forced to retract two papers it had published about stem cells. The articles had been submitted by a South Korean team led by Hwang Woo-Suk. Peer reviewers, as well as the editors, had failed to detect the fraud.

In general, peer reviewers, themselves researchers pressed for time, don't try to re-create experiments and rarely ask to see the raw data that supports a paper's conclusions. While peer review is expected to separate the wheat from the chaff, it's "slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud," summed up one critic in BMJ, the British medical journal, in 1997.

"There's a lot of discussion [in the scientific community] about how peer review doesn't work," Mr. Surridge says. "It's not a great way to decide [what to publish]. It's just the only way we have at the moment."

PLoS ONE takes a different tack. While articles receive a basic screening, they don't have to attain the standard of representing groundbreaking work in order to be published. An article only has to be based on solid science. The idea is that the more valid research is published, the better, as it contributes to an online database.

"If it is science, [if] it is well done, [and if] it provides a valuable contribution to scientific literature, we can publish it," Surridge says. He expects about two-thirds of those papers submitted to PLoS ONE to be accepted.

Since its launch Dec. 20, PLoS ONE has published well over 100 papers and expects to publish 15 to 20 more per week. Readers access the articles for free. PLoS ONE pays its way by charging authors $1,250 to publish an article. While that might seem a barrier to publication, Surridge says most research is financed by grants or large institutions, meaning individual scientists rarely have to pay themselves. But just in case, PLoS ONE is waiving the fee for any authors who request it.

Moshe Prisker and Nikita Bernstein are taking another approach, using the Web's ability to deliver video easily with JoVE. While scientific concepts can be very simple, "the actual doing of the [laboratory] experiments is very difficult," says Dr. Prisker, a neural-stem-cell researcher at Massachusetts General Hospital in Boston.

Traditionally, he says, scientists have resorted to wandering from lab to lab asking, "Do you know how to do this or do that? They're basically asking someone with previous experience to show them."

In its first three months, JoVE has posted 18 videos ranging from five to 15 minutes showing techniques such as "Nuclear transfer in mouse oocytes." They get a modest vetting from scientists in the field to make sure they are sound. The site is free to view and charges nothing to post videos. Prisker says that he and Mr. Bernstein, both volunteers, hope that the journal will someday pay for itself through ads from manufacturers of lab equipment (the "Google model," he says).

Other journals are beginning to employ video in some articles, but JoVE is the first to make video images the primary means of conveying information. Brief articles, voice-overs, or captions accompany the moving images.

"Video gives you this ability for unambiguous representation of experiments," Prisker says. "I'm sure that video publication will become a significant force [in online journals]." Nearly all of the early reaction to the fledgling journal has been positive. Visitors to the JoVE website say, "This should have been done long ago," Prisker says.

Since the early days of the Web, observers have speculated that scientists might simply post new research on their own or in communal websites and let search engines find it, thereby bypassing the peer-reviewed journals altogether. If the research proves valuable, other sites will link to it, and the results would be "published" far faster than waiting for a journal to accept them.

Already, an online database called arXiv (www.arXiv.org), hosted by Cornell University in Ithaca, N.Y., contains more than 400,000 scientific papers posted by their authors without peer review. (The papers often appear later in peer-reviewed journals). Its comprehensiveness makes arXiv (pronounced "archive") a valuable tool, Gerstein, the Yale researcher, says. If someone claims to make a new discovery, anyone can search this database and say, "No, you didn't. It's in the arXiv."

Nonetheless, Gerstein says he thinks scientific journals, and some kind of peer review, will be around for a long time. Publishing in prestigious journals is "deeply intertwined with [scientists'] reputations and their promotions," he says. "You still want to get the stamp of approval of a journal."

- - -

See also:

Web journals threaten peer-review system
Instead of having a group of hand-picked scholars review research in secret before publication, a growing number of Internet-based journals are publishing studies with little or no scrutiny by the authors' peers. It's then up to rank-and-file researchers to debate the value of the work in cyberspace. The Web journals are threatening to turn on its head the traditional peer-review system that for decades has been the established way to pick apart research before it's made public.

PrintPrintable Version

2 Comments

Sid Bertram comments (by email):

I am doubtful the new policy will help the situation much. I sent a copy of my paper to the Cal Tech Alumni association where I got my B.S. (with honor) in 1938 and finally got a reply from a physicist there. It was the following:

“I well remember the paper by Dr. Bertram since I scanned it and noted what appeared to be an inconsistency with the principles of relativity. I did not have the time to do a proper analysis to be certain that my immediate impression was correct."

While I was able to correspond with him a bit, I was unable to get him to read it again.

On the other hand, from a retired professor of electrical engineering who I had known pretty well:

“I finally had a chance to read your paper carefully, I cannot disagree with your treatment or your model and I believe it should be a useful approach to understanding the force-charge relations in relativistic situations.

Although I believe your paper should be published, I would anticipate some difficulty in getting it accepted because of the way most referees have been taught to think. Nonetheless, I think it should be submitted because it would be a definite contribution.”

I don't think they would let us choose the reviewers. I hope I'm unduly pessimistic.

Sid Bertram

Here are some facts about my experience with technical paper submissions. In 1994, I submitted a paper on a new electromagnetic model of the hydrogen atom to the IEEE. They claimed to be interested, but they could not find anyone willing to review the paper. Since I felt that timing is very important, I decided to publish it myself. The result was my first book "The Secret of Gravity".

Subsequent studies substantiated earlier results that electromagnetic field waves must bend with velocity. This result tends to support the concept of an ether in the universe, so I submitted a paper to the IEE in England. They also could not find anyone qualified to review the paper and advised me to submit it to a physics organization. I contacted the APS about my paper, but they did not respond. So I wrote my second and third books.

The next situation was much more favorable. I had been studying Planck's Columbia Lectures and recognized the resemblance of his energy state equation to the measured properties of electronic thermal noise. Several papers on this subject were published in the ACS magazine "Chemical Innovation". Then another paper on electromagnetic radiation, "A Different Picture of Radiation", was presented at the IEEE Antennas and Propagation Society International Symposium 2003 in which it was shown that electromagnetic waves can move faster than the speed of light (but not in the radial direction).

My latest submission, "Magnetic Effects of an Electron Moving in the Near Field
of a Proton" was accepted by the IEEE Magnetics Society, but was subsequently rejected by two reviewers (after waiting for over a year for them to find reviewers). The initial comment was that the magnetic field of an atom cannot be assessed by "electric current" since there is but a single electron. However, I pointed out that this was exactly what Bohr did in calculating the "Bohr magneton". The final reason for rejection was that no evidence was presented that the material in my paper correlates with quantum theory. However, I did present certain correlating facts, although, as I pointed out, to present full correlation would take several papers. The rejection was not open to appeal, and I was denied the opportunity to challenge the reviewers.

It seems that a paper on anything new or controversial stands little chance of being accepted. If the subject of the paper crosses the boundary between disciplines, then the chance of finding a qualified reviewer is slim.

Looking at the other side of the problem, there are writers who have come up with some theories that have little or no correlation to reality and have but slight knowledge of the subject upon which they speak. This does present a problem to those of us who put a lot of effort into our investigations, since a paper of this type may cause serious objections by readers and a bias towards new ideas. Even Professor Planck had difficulty in getting his blackbody radiation theory accepted, and there are those scientists today who still do not accept it, even though he is generally regarded as the founder of quantum theory. The newer, but not well understood, methods of quantum mechanics is obviously another complicating factor for the subject of my studies.


Leave a comment

Sepp


Receive updates

Email updates for new articles

Enter your Email