> From: Alexander Berezin <berezin_at_MCMAIL.CIS.MCMASTER.CA>
> To: Canadian Association for Responsible Research Funding
> <CARRF_at_YORKU.CA>
>
> Published in "University Affairs" (Association
> of Universities and Colleges of Canada),
> Vol. 41 (#1), January 2000, page 5.
>
> PEER REVIEW OR PUBLIC REVIEW?
>
> In heated discussions on the recent initiative of
> the National Institutes of Health to launch free access
> through PubMed ("NIH plan attracts praise and criticism",
> University Affairs, November 1999), the concerns about
> the alleged quality of peer review of electronic
> publications appear to be grossly overstated. In fact,
> electronic format allows for a relatively easy offset of
> the deficiencies of the traditional pre-publication peer
> review.
>
> Among the most serious flaws of the peer review
> system is the anonymity of the reviewers (and hence lack
> of any real accountability on their part), ample
> possibilities for the formation of "editorial cabals"
> which seize control of key research journals, and the
> general conservatism of the peer review system.
>
> In the Internet age, printed research journals
> have lost their major purpose of being vehicles of
> information. Their prime role is now to buttress the
> prestige structure of the scientific community. In
> opposing the transition from paper-bound to electronic
> journals, the argument that peer review is needed to
> maintain quality is primarily put forward to divert
> attention from the true fear of the power-controlling
> research elite.
>
> That fear is a forthcoming collapse (or at the
> very least, radical adjustment) of the fictitious
> prestige system of the existing "established" journals.
> We all know that modern science is largely not what you
> publish, but on what paper you manage to get it printed.
> This trend becomes more and more pronounced. The rat
> race to publish in a few top research tabloids has
> reached pathological proportions. Recent multimillion
> dollar litigation between Immunex and Cistron biomedical
> research groups on the matter of the alleged theft of
> ideas during peer review of an article submitted to
> Nature magazine is just one of many exhibits demonstrating
> how easily anonymous peer review can get corrupted.
>
> In contrast to this, replacement of paper journals
> by electronic repositories allows for the fast and
> interactive monitoring of the quality, importance and
> originality of posted articles through the system of
> added comments from the readership. This is a much more
> efficient system for detecting erroneous, trivial or
> plagiarized work than the pre-publication peer review.
> What we need is a "publish all" strategy, with only the
> most basic check on obscene and gratuitous material.
>
> The argument that publishing without peer review
> will result in a flood of garbage does not hold water.
> The opposite is likely to happen. Without the game of
> "journal prestige", scientists will likely publish less,
> not more: only when they really have something new to say,
> not to score points for publication in "prestigious"
> journals.
>
> The calibre of researchers would then be determined,
> not by how many papers they have published in this-or-
> that prestigious journal, but by what they have ACTUALLY
> DISCOVERED. Any electronically published article can be
> openly criticized and questioned. With the exception of
> fraudulent researchers, a system of free added comments
> threatens no one.
>
> Free and uncontrolled access to science journals
> is technically possible and its costs are insignificant
> in comparison with the prime research expenses. Full
> texts of all posted (hence, published) scientific
> articles should be available for free to anyone, anytime,
> anyplace. Period. And as for the costs, one can easily
> estimate that free electronic access to all scientific
> articles for the next 100 years will amount to only a
> tiny fraction of the cost of the recent war in Kosovo.
> Go for it.
>
> Alexander A. Berezin.
> Dr. Berezin is a professor of engineering physics
> at McMaster University.
> -----------------------------------------------------
Professor Berezin makes a number of valid points about problems with
the peer review system (both in publication and funding), a system
which was not devised by systematic empirical testing and optimization,
but one that evolved across many decades of fumbling practise.
The peer review system should be studied, and ways should be sought to
improve it. However, there is neither any sense nor any justification
in COUPLING in any way (1) the mission of peer-review reform, which
will require careful, systematic testing to find out what will make
things better rather than worse, with (2) the mission of freeing the
current peer-reviewed literature from the access barriers of (a) paper
and (b) paper's cost-recovery mechanism (Subscription, Site-License,
Pay-Per-View, S/L/P).
There is a simple reason for this: no peer-review reforms to date have
been tested and validated empirically, whereas the benefits of freeing
the peer-reviewed literature have a face-validity that has already been
demonstrated empirically <
http://xxx.lanl.gov/cgi-bin/show_weekly_graph>
Hence it makes no sense whatsoever to link the fates of these two
independent desiderata, one purely speculative, the other already
demonstrated.
(This is not to say that the online medium will not be useful in
optimizing quality-control, whether through classical peer review or
its empirically validated successor; but there is no basis for coupling
them NOW.)
This theme of decoupling peer review reform from paper medium reform
has already been much discussed in this Forum and elsewhere
<
http://www.nih.gov/welcome/director/ebiomed/com0509.htm#harn45>
<
http://library.caltech.edu/publications/ScholarsForum/042399sharnad.htm>
See also references at the end of this commentary.
Some brief quote/comments on Professor Berezin's article follow:
> Among the most serious flaws of the peer review
> system is the anonymity of the reviewers (and hence lack
> of any real accountability on their part)
This is a conjecture that many have made, but no one has ever backed up
with any evidence that it is true. What would be required would be an
independent measure of quality, and evidence that the option of referee
anonymity lowers rather than raises it.
Note that in a democracy we vote anonymously too (does it diminish the
quality of our choices -- such as it is)? The objection to anonymity is
usually made from the viewpoint of the aggrieved author, unable to
confront his judges.
But, first of all, there IS accountability: to the Editor. The Editor
knows the identity of both the author and the referees; moreover, he is
answerable for his editorial decisions (the journal's quality and impact
are among the indicators to which those decisions are ultimately
answerable, as are the journal's readership, and, yes, its authorship
too).
This can all stand with a good deal of improvement, I agree. But the
apriori assumption that referee anonymity is the culprit, and open
refereeing the remedy, is pure speculation. As food for thought, what
about competent junior researchers refereeing the papers of incompetent
(but influential) senior researchers? Will they dare to do a forthright
critique if their own future careers and grants are put at risk by it?
Yes, there could be abuse in the other direction too, but that's where
it is the EDITOR's competence that is the real lynchpin of it all. (And
that's probably the factor that could use the most empirical scrutiny in
search of improvement.)
Similar points could be raised about another armchair-based
recommendation: author anonymity. No one has shown whether blind review
increases or decreases the validity of refereeing; steps have merely
been taken on the basis of assumptions, and the consequences have not
been analyzed at all: Yes, the author's identity can elicit positive or
negative biasses in some reviewers, sometimes (it's the Editor's job to
see to it that that does not happen), but overall, might a referee not
be better off evaluating a paper in context, relative to what an author
has done before? These are empirical questions, not a priori ones.
> In opposing the transition from paper-bound to electronic
> journals, the argument that peer review is needed to
> maintain quality is primarily put forward to divert
> attention from the true fear of the power-controlling
> research elite.
Without prejudice as to whether or not there really exists such an elite
in the welter of journals that exist, there are plenty of reasons for
opposing online-only journals (untested habits, mainly) and even more
for opposing free online-only journals (interests vested in preserving
current S/L/P revenue streams) without ever broaching the question of
whether peer review should be tampered with in any way. Again, the two
variables are independent.
> We all know that modern science is largely not what you
> publish, but on what paper you manage to get it printed.
That would imply a rather peculiar causal interpretation of, say,
impact factors, namely, that authors cite articles because of the
prestige of the journal they appear in, rather than the quality and
relevance of the article/author, and that the prestige of the journals
is in turn owing to their names and associated elite, and not to their
impact factors. Surely this causality needs to be tested empirically,
before we draw any strong conclusions; it does, on the face of it, make
scientific progress, such as it is, look rather surprising, though.
> anonymous peer review can get corrupted.
Indeed it can. How often does it? And what are the alternatives? These
are the questions we must ask. There are no answers yet.
> In contrast to this, replacement of paper journals
> by electronic repositories allows for the fast and
> interactive monitoring of the quality, importance and
> originality of posted articles through the system of
> added comments from the readership. This is a much more
> efficient system for detecting erroneous, trivial or
> plagiarized work than the pre-publication peer review.
I will not comment on this as it has been so much discussed in this
forum -- except to remind readers that on this particular point I may
be better placed to make an educated guess, having umpired both
anonymous peer review and open peer commentary for the past two
decades, and my own experience has been that peer commentary is an
excellent SUPPLEMENT TO but certainly no SUBSTITUTE FOR peer review. I
would not like to wake up one morning and find that, whilst wearing my
active researcher's hat, I had to try to wade through the same raw,
unfiltered sludge that I must wade through when I am wearing my editor's
hat and contending with new submissions.
<
http://www.princeton.edu/~harnad/bbs/index.html>
<
http://www.princeton.edu/~harnad/psyc.html>
In fact, I am convinced that there is only one thing left to be paid
for, once the paper literature has gone online and become free to the
reader, and that is the Quality-Control and Certification that comes
from peer review.
> The argument that publishing without peer review
> will result in a flood of garbage does not hold water.
> The opposite is likely to happen. Without the game of
> "journal prestige", scientists will likely publish less,
> not more: only when they really have something new to say,
> not to score points for publication in "prestigious"
> journals.
Human nature being what it is, I doubt it. But don't you think this is a
rather bold prediction to make without any data whatsoever? (There ARE
data that give some indication of what information/misinformation might
look like if it were answerable only to public comments: See the chat
groups on the Global Graffiti Board for Trivial Pursuit, USENET.)
> Any electronically published article can be
> openly criticized and questioned.
But who would have the time or the inclination to patrol the Global
Graffiti Board on our behalf? -- and are they really the ones we want to
trust, rather than the peers selected for their expertise BY Editors,
answerable TO those Editors, who are in turn answerable to us?
Is Usenet the basis on which one would want a doctor to decide on a
cure for the disease of a loved one?
> Full texts of all posted (hence, published) scientific
> articles should be available for free to anyone, anytime,
> anyplace.
Can we free the journal literature from paper and its costs first, and
then worry about whether (and how) to free it from peer review?
REFERENCES
Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in
scientific quality control, New York: Cambridge University Press.
Harnad, S. (1984) Commentaries, opinions and the growth of scientific
knowledge. American Psychologist 39: 1497 - 1498.
Harnad, S. (1985) Rational disagreement in peer review. Science,
Technology and Human Values 10: 55 - 62.
Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A
difficult balance: Peer review in biomedical publication.)
Nature 322: 24 - 5.
Harnad, S. (1996) Implementing Peer Review on the Net:
Scientific Quality Control in Scholarly Electronic Journals. In:
Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic
Frontier. Cambridge MA: MIT Press. Pp. 103-118.
http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
Harnad, S. (1998) Learned Inquiry and the Net:
The Role of Peer Review, Peer Commentary and Copyright.
Learned Publishing 4(11): 283-292
http://citd.scar.utoronto.ca/EPub/talks/Harnad_Snider.html
Harnad, S. (1998) The invisible hand of peer review. Nature [online] (5
Nov. 1998) <
http://helix.nature.com/webmatters/invisible/invisible.html
Harnad, S. (1999) Free at Last: The Future of Peer-Reviewed Journals.
D-Lib Magazine 5(12) December 1999
http://www.dlib.org/dlib/december99/12harnad.html
--------------------------------------------------------------------
Stevan Harnad harnad_at_cogsci.soton.ac.uk
Professor of Cognitive Science harnad_at_princeton.edu
Department of Electronics and phone: +44 23-80 592-582
Computer Science fax: +44 23-80 592-865
University of Southampton
http://www.ecs.soton.ac.uk/~harnad/
Highfield, Southampton
http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM
NOTE: A complete archive of this ongoing discussion of "Freeing the
Refereed Journal Literature Through Online Self-Archiving" is available
at the American Scientist September Forum (98 & 99):
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html
Received on Wed Feb 10 1999 - 19:17:43 GMT