Peer Review - Nature Initiates Trial and Debate

- - -

Here is how Philip Campbell PhD, Editor-in-Chief of Nature, describes the experiment:

Nature's peer review trial

Peer review is the bedrock of scientific publication (for Nature's position on peer review, see our Guide to Authors). It is widely considered essential for improving submitted papers and enhancing the credentials of scientists as well as those of the journals in which they choose to publish.

But, like any process, peer review requires occasional scrutiny and assessment. Has the Internet brought new opportunities for journals to manage peer review more imaginatively or by different means? Are there any systematic flaws in the process? Should the process be transparent or confidential? Is the journal even necessary, or could scientists manage the peer review process themselves?

Nature's peer review process has been maintained, unchanged, for decades. We, the editors, believe that the process functions well, by and large. But, in the spirit of being open to considering alternative approaches, we are taking two initiatives: a web debate and a trial of a particular type of open peer review.

The trial will not displace Nature's traditional confidential peer review process, but will complement it. From 5 June 2006, authors may opt to have their submitted manuscripts posted publicly for comment.

Any scientist may then post comments, provided they identify themselves. Once the usual confidential peer review process is complete, the public 'open peer review' process will be closed. Editors will then read all comments on the manuscript and invite authors to respond. At the end of the process, as part of the trial, editors will assess the value of the public comments.

At the close of the trial, we will assess the value of public comments overall as well as the practicalities of their inclusion on a longer-term basis. We will publish an account of the trial and our conclusions.

For further details about the trial for authors and reviewers, please go to the question and answer page provided. Further general questions not answered by this list can be directed to the editors. Questions from authors and referees about particular manuscript submissions should be sent to the editor who is handling the manuscript.

(... on Nature.com)

Update December 2006:

Science journal 'Nature' abandons peer review experiment
Citing a lack of participation, the British journal Nature said it was ditching a closely watched online experiment that allowed scientists to comment on their peers' research before publication. The four-month trial, which began in June, was aimed at democratizing the peer review process, a time-honored tradition in which a group of select scholars critique scientific manuscripts and decide what appears in print.

- - -

Although Nature ended its experiment, the end of peer review as we know it seems to be steadily approaching. Here is an interesting article that appeared in USA Today - January 2007:

Is this the end of the scholarly journal?

By Gregory M. Lamb, The Christian Science Monitor

Scientific advances sometimes come as lightning flashes of inspiration. But when scientists sit down to record and take credit for what they've found, they still use much the same method they have for decades - an article published in a scholarly journal.

But science's hidebound traditions are changing. The Internet has opened up new forms of publishing in which anyone in the world can find and read a scientific paper. And papers themselves are becoming more interactive, leading readers to the underlying data, videos, and discussions that augment their value. With blogs and e-books providing easy means of self-publishing, some observers are speculating that scholarly journals and their controversial system of peer reviews may not be needed at all.

"The traditional journal publishing medium we've grown used to really needs to evolve and change because that's not the way people are accessing information," says Mark Gerstein, a professor of biomedical informatics at Yale University in New Haven, Conn. Dr. Gerstein cowrote an article, "The Death of the Scientific Paper," which appeared last year on The-Scientist.com, an online science magazine.

If the hopes of innovators bear fruit, scientific advances will come ever more quickly as online publishing makes past research easier to access and share widely.

Two new scientific publications, both available only online, may signal what's ahead. The PLoS ONE (plosone.org), a journal begun by the Public Library of Science (PLoS) last month, aims to put as many new scientific articles as possible on the Internet to be read by anyone, free of charge. The Journal of Visualized Experiments, or JoVE (myjove.com), is a kind of YouTube for researchers. It operates on the theory that a short video showing how an experiment is done is better than thousands of words that attempt to describe it.

At PLoS ONE, which aspires to be a general science journal along the lines of Science and Nature, the papers themselves are only a starting point. Readers can annotate, comment on, and critique the findings: Their contributions become permanently attached to the original article. At least one commentator has likened this process to a kind of "electronic Talmud," in which the original document receives elaborate commentary and discussion that over time adds greatly to its value.

In coming months, says Chris Surridge, the managing editor of PLoS ONE, readers also will be able to rate papers on their quality, such as how surprising or groundbreaking the results were - much in the way Netflix subscribers rate movies they rent using one- to five-star ratings. In this sense, PLoS ONE is moving toward a Web 2.0 model, which focuses on user-generated content strategies already used by websites such as Digg.com, Slashdot.org, or Amazon.com.

For years, traditional "peer review" has come under fire. A jury of three experts, the peer reviewers, assess each article and recommend only those that they feel represent the most significant new work. At many elite scientific journals, fewer than 10% of the articles submitted are accepted. Many of the rejected articles eventually travel down the "food chain" to be published in a plethora of less prestigious (and less noticed) specialty journals.

A year ago, the respected U.S. journal Science was forced to retract two papers it had published about stem cells. The articles had been submitted by a South Korean team led by Hwang Woo-Suk. Peer reviewers, as well as the editors, had failed to detect the fraud.

In general, peer reviewers, themselves researchers pressed for time, don't try to re-create experiments and rarely ask to see the raw data that supports a paper's conclusions. While peer review is expected to separate the wheat from the chaff, it's "slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud," summed up one critic in BMJ, the British medical journal, in 1997.

"There's a lot of discussion [in the scientific community] about how peer review doesn't work," Mr. Surridge says. "It's not a great way to decide [what to publish]. It's just the only way we have at the moment."

PLoS ONE takes a different tack. While articles receive a basic screening, they don't have to attain the standard of representing groundbreaking work in order to be published. An article only has to be based on solid science. The idea is that the more valid research is published, the better, as it contributes to an online database.

"If it is science, [if] it is well done, [and if] it provides a valuable contribution to scientific literature, we can publish it," Surridge says. He expects about two-thirds of those papers submitted to PLoS ONE to be accepted.

Since its launch Dec. 20, PLoS ONE has published well over 100 papers and expects to publish 15 to 20 more per week. Readers access the articles for free. PLoS ONE pays its way by charging authors $1,250 to publish an article. While that might seem a barrier to publication, Surridge says most research is financed by grants or large institutions, meaning individual scientists rarely have to pay themselves. But just in case, PLoS ONE is waiving the fee for any authors who request it.

Moshe Prisker and Nikita Bernstein are taking another approach, using the Web's ability to deliver video easily with JoVE. While scientific concepts can be very simple, "the actual doing of the [laboratory] experiments is very difficult," says Dr. Prisker, a neural-stem-cell researcher at Massachusetts General Hospital in Boston.

Traditionally, he says, scientists have resorted to wandering from lab to lab asking, "Do you know how to do this or do that? They're basically asking someone with previous experience to show them."

In its first three months, JoVE has posted 18 videos ranging from five to 15 minutes showing techniques such as "Nuclear transfer in mouse oocytes." They get a modest vetting from scientists in the field to make sure they are sound. The site is free to view and charges nothing to post videos. Prisker says that he and Mr. Bernstein, both volunteers, hope that the journal will someday pay for itself through ads from manufacturers of lab equipment (the "Google model," he says).

Other journals are beginning to employ video in some articles, but JoVE is the first to make video images the primary means of conveying information. Brief articles, voice-overs, or captions accompany the moving images.

"Video gives you this ability for unambiguous representation of experiments," Prisker says. "I'm sure that video publication will become a significant force [in online journals]." Nearly all of the early reaction to the fledgling journal has been positive. Visitors to the JoVE website say, "This should have been done long ago," Prisker says.

Since the early days of the Web, observers have speculated that scientists might simply post new research on their own or in communal websites and let search engines find it, thereby bypassing the peer-reviewed journals altogether. If the research proves valuable, other sites will link to it, and the results would be "published" far faster than waiting for a journal to accept them.

Already, an online database called arXiv (www.arXiv.org), hosted by Cornell University in Ithaca, N.Y., contains more than 400,000 scientific papers posted by their authors without peer review. (The papers often appear later in peer-reviewed journals). Its comprehensiveness makes arXiv (pronounced "archive") a valuable tool, Gerstein, the Yale researcher, says. If someone claims to make a new discovery, anyone can search this database and say, "No, you didn't. It's in the arXiv."

Nonetheless, Gerstein says he thinks scientific journals, and some kind of peer review, will be around for a long time. Publishing in prestigious journals is "deeply intertwined with [scientists'] reputations and their promotions," he says. "You still want to get the stamp of approval of a journal."

- - -

See also:

Web journals threaten peer-review system
Instead of having a group of hand-picked scholars review research in secret before publication, a growing number of Internet-based journals are publishing studies with little or no scrutiny by the authors' peers. It's then up to rank-and-file researchers to debate the value of the work in cyberspace. The Web journals are threatening to turn on its head the traditional peer-review system that for decades has been the established way to pick apart research before it's made public.