Stevan, Enclosed below is a draft of a short note about peer review that
was solicited for the 2nd edition of "Peer Review in Health Sciences."
Any comments you might have would be greatly appreciated.
Best regards, Andrew
Peer and non-peer review
Andrew Odlyzko
Digital Technology Center
University of Minnesota
http://www.dtc.umn.edu/~odlyzko
Fears about possible damage to the peer review system are slowing down the
evolution of scholarly communication, and in particular the development
of freely accessible article archives. I am convinced that these fears
is unjustified. Although the peer review system will change substantially
with the spread of such archives, it will change for the better.
A good overview of the history and current state of the peer review
system is provided by the book [1]. This system is really a collection
of many different systems, of varying effectiveness. They guarantee
neither correctness nor novelty of the results, even among the most
selective and prestigious journals. However, traditional peer review
(with anonymous referees evaluating submissions to a journal) does
perform a valuable screening function. Still, it is just a part of
the entire communication system, and evaluation of the value of an
article is never truly complete, as sometimes historians will revisit
this question centuries after publication. It is the presence of such
self-correcting features in the entire scholarly communication system
that makes the deficiencies of the current peer review system tolerable.
However, it is natural to expect evolution to occur.
In the Gutenberg era of print journals, placing heavy reliance on
traditional peer review was sensible. Printing and distributing journals
was very expensive. Furthermore, providing additional feedback after
publication was hard and slow. Therefore it was appropriate to devote
considerable attention to minimizing the volume of published material,
and making sure it was of high quality. With the development of more
flexible communication systems, especially the Internet, we are moving
towards a continuum of publication. I have argued, starting with [2],
that this requires a continuum of peer review, which will provide feedback
to scholars about articles and other materials as they move along the
continuum, and not just in the single journal decision process stage.
We can already see elements of the evolving system of peer review in
operation.
Many scholars, including Stevan Harnad [3], one of the most prominent
proponents of open access archives, argue for a continuing strong role
for the traditional peer review system at the journal level. I have no
doubt that this system will persist for quite a while, since sociological
changes in the scholarly arena are very slow [4]. However, I do expect
its relative importance to decline. The reason is that there is a
continuing growth of other types of feedback that scholars can rely on.
This is part of the general trend (described in [5]) in which traditional
journals are continuing as before, but the main action is in novel and
often informal modes of communication that are growing much more rapidly.
The growing flood of information does require screening. Some of this
reviewing can be done by non-peers. Indeed, some of it has traditionally
been done by non-peers, for example in legal scholarship, where U.S. law
reviews are staffed by students. The growing role of interdisciplinary
research might lead to a generally greater role for non-peers in reviewing
publications. However, in most cases only peers are truly qualified to
review technical results. However, peer evaluations can be obtained,
and increasingly are being obtained, much more flexibly than through the
traditional anonymous journal refereeing process. Some can come from
use of automated tools to harvest references to papers, in a much more
flexible and comprehensive way than the Science Citation Index provided
in the old days. Other, more up-to-date evaluations, can be obtained
from a variety of techniques, such as those described in [5].
An example of how evolving forms of peer review function is provided by
the recent proof that testing whether a natural number is prime (that
is, divisible only by 1 and itself) can be done fast. (The technical
term is in "polynomial time.") This had been an old and famous open
problem of mathematics and computer science. On Sunday, August 4, 2002,
Maninda Agrawal, Neeraj Kayal, and Nitin Saxena of the Indian Institute
of Technology in Kanpur sent out a paper with their astounding proof of
this result to several of the recognized experts on primality testing.
(Their proof was astounding because of its unexpected simplicity.)
Some of these experts responded almost right away, confirming the validity
of the proof. On Tuesday, August 6, the authors then posted the paper
on their Web site and sent out email announcements. This prompted many
additional mathematicians and computer scientists to read the paper, and
led to extensive discussions on online mailing lists. On Thursday, August
8, the New York Times carried a story announcing the result and quoting
some of the experts who had verified the correctness of the result.
Review by peers played a central role in this story. The authors first
privately consulted known experts in the subject. Then, after getting
assurance they had not overlooked anything substantial, they made their
work available worldwide, where it attracted scrutiny by other experts.
The New York Times coverage was based on the positive evaluations of
correctness and significance by those experts. Eventually they did
submit their paper to a conventional journal, where it will undoubtedly
undergo conventional peer review, and be published. The journal version
will probably be the main one cited in the future, but will likely have
little influence on the development of the subject. Within weeks of the
distribution of the Agrawal-Kayal-Saxena article, improvements on their
results had been obtained by other researchers, and future work will be
based mainly on those. Agrawal, Kayal, and Saxena will get proper credit
for their breakthrough. However, although their paper will go through
the conventional journal peer review and publication system, that will
be almost irrelevant for the intellectual development of their area.
One can object that only potentially breakthrough results are likely
to attract the level of attention that the Agrawal-Kayal-Saxena result
attracted. But that is not a problem. It is only the most important
results that require this level of attention and at this rapid a rate.
There will be a need for some systematic scrutiny of all technical
publications, to ensure that the literature does not get polluted to
erroneous claims. However, we should expect a much more heterogeneous
system to evolve, in which many of the ideas mentioned in [2] will play
a role. For example, the current strong prohibition of simultaneous
publication in multiple journals is likely to be discarded as another
relic of the Gutenberg era where print resources were scarce. Also,
we are likely to see separate evaluations of significance and correctness.
This note is a personal perspective on how peer review is likely to evolve
in the future. It is based primarily on my experience in area such as
mathematics, physics, computing, and some social sciences. However,
I believe there is nothing special about those areas. Although health
sciences have moved towards electronic publishing more slowly than the
fields I am familiar with, I do not see much that is special about
their needs. In particular, I believe that the frequently voiced
concerns about need for extra scrutiny of research results that might
affect health practices are a red herring. Yes, decision about medical
procedures or even diet should be based on solidly established research.
However, the extra levels of scrutiny are more likely to be obtained by
more open communication and review systems than we have today.
[1] F. Godlee and T. Jefferson, eds., "Peer Review in Health Sciences,"
BMJ Books, 1999.
[2] A. M. Odlyzko, Tragic loss or good riddance? The impending demise
of traditional scholarly journals, Intern. J. Human-Computer Studies
(formerly Intern. J. Man-Machine Studies) 42 (1995), pp. 71-122.
Available online at <
http://www.dtc.umn.edu/~odlyzko/doc/eworld.html>.
[3] S. Harnad, The invisible hand of peer review, Exploit
Interactive, issue 5, April 2000. Available online at
<
http://www.exploit-lib.org/issue5/peer-review/> and
<
http://www.ecs.soton.ac.uk/~harnad/nature2.html>.
[4] A. M. Odlyzko, The slow evolution of electronic publishing, in
"Electronic Publishing '97: New Models and Opportunities," A. J. Meadows
and F. Rowland, eds., ICCC Press, 1997, pp. 4-18. Available online at
<
http://www.dtc.umn.edu/~odlyzko/doc/eworld.html>.
[5] A. M. Odlyzko, The rapid evolution of scholarly communication,
Learned Publishing, 15(1) (Jan. 2002), pp. 7-19. Available online
at <
http://www.catchword.com/alpsp/09531513/v15n1/contp1-1.htm> and
<
http://www.dtc.umn.edu/~odlyzko/doc/eworld.html>.
Received on Mon Nov 04 2002 - 14:21:24 GMT