On Sun, 12 Mar 2000, Andrew Kenneth Fletcher wrote:
> I have a real problem with the current Peer Review System. It is biased
> towards in-house publications and outsiders are ignored.
>
> I had an idea to set up a new newsgroup titled "Peer Review Sci" I am
> certain that it would attract many professional contributors, who would
> normally have been ignored by publishers and therefore provide independent
> researchers with an unbiased review of their work.
> It would also be a far better way to make sure that nothing false arrives in
> print, because it would be an open peer review system and anyone
> contributing either a paper or a review of a paper would be open to comment
> from other reviewers. This would generate a tremendous amount of new science
> and encourage the people with the ideas to come forward.
>
> What say ye to this?
The notion of replacing peer review by some form of open commentary has
been proposed many times, in this Forum and elsewhere (and it is being
experimented with by several sites on the Web). See the other threads on
this in this Forum (1999, 1999, & 2000) and:
Harnad, S. (1998) The invisible hand of peer review. Nature
[online] (c. 5 Nov. 1998)
http://helix.nature.com/webmatters/invisible/invisible.html
Longer version below to appear in Exploit Interactive
<
http://www.exploit-lib.org/>:
http://www.ecs.soton.ac.uk/~harnad/nature2.html
Here are some relevant excerpts from the above:
Self Policing?
Alternatives have of course been proposed, but to propose is
not to demonstrate viability. Most proposals have envisioned
weakening the constraints of classical peer review in some way
or other. the most radical way being to do away with it
altogether: Let authors police themselves; let every submission
be published, and let the reader decide what is to be taken
seriously. This would amount to discarding the current
hierarchical filter -- both its active influence, in directing
revision, and its ranking of quality and reliability to guide
the reader trying to navigate the ever-swelling literature
(Hitchcock et al. 2000).
There is a way to test our intuitions about the merits of this
sort of proposal a priori, using a specialist domain that is
somewhat more urgent and immediate than abstract "learned
inquiry"; then if we are not prepared to generalise this
intuitive test's verdict to scholarly/scientific research in
general, we really need to ask ourselves how seriously we take
the acquisition of knowledge: If someone near and dear to you
were ill with a serious but potentially treatable disease,
would you prefer to have them treated on the basis of the
refereed medical literature or on the basis of an unfiltered
free-for-all where the distinction between reliable expertise
and ignorance, incompetence or charlatanism is left entirely to
the reader, on a paper by paper basis?
A variant on this scenario is currently being tested by the
British Medical Journal
<
http://www.bmj.com/cgi/shtml/misc/peer/index.shtml>, but
instead of entrusting entirely to the reader the quality
control function performed by the referee in classical peer
review, this variant, taking a cue from some of the
developments and goings-on on both the Internet and Network TV
chat-shows, plans to publicly post submitted papers unrefereed
on the Web and to invite any reader to submit a commentary;
these commentaries will then be used in lieu of referee reports
as a basis for deciding on formal publication.
Expert Opinion or Opinion Poll?
Is this peer review? Well, it is not clear whether the
self-appointed commentators will be qualified specialists (or
how that is to be ascertained). The expert population in any
given speciality is a scarce resource, already overharvested by
classical peer review, so one wonders who would have the time
or inclination to add journeyman commentary services to this
load on their own initiative, particularly once it is no longer
a rare novelty, and the entire raw, unpoliced literature is
routinely appearing in this form first. Are those who have
nothing more pressing to do with their time than this really
the ones we want to trust to perform such a critical QC/C
function for us all?
And is the remedy for the possibility of bias or incompetence
in referee-selection on the part of editors really to throw
selectivity to the winds, and let referees pick themselves?
Considering all that hangs on being published in refereed
journals, it does not take much imagination to think of ways
authors could manipulate such a public-polling system to their
own advantage, human nature being what it is.
Peer Commentary vs. Peer Review
And is peer commentary (even if we can settle the vexed "peer"
question) really peer review? Will I say publicly about someone
who might be refereeing my next grant application or tenure
review what I really think are the flaws of his latest raw
manuscript? (Should we then be publishing our names alongside
our votes in civic elections too, without fear or favour?) Will
I put into a public commentary -- alongside who knows how many
other such commentaries, to be put to who knows what use by who
knows whom -- the time and effort that I would put into a
referee report for an editor I know to be turning specifically
to me and a few other specialists for our expertise on a
specific paper?
If there is anyone on this planet who is in a position to
attest to the functional difference between peer review and
peer commentary (Harnad 1982, 1984), it is surely the author of
the present article, who has been umpiring a peer-reviewed
paper journal of Open Peer Commentary (Behavioral and Brain
Sciences [BBS] <
http://www.princeton.edu/~harnad/bbs.html>,
published by Cambridge University Press) for over 2 decades
(Harnad 1979), as well as a brave new online-only journal of
Open Peer Commentary, likewise peer-reviewed (Psycoloquy,
sponsored by the American Psychological Association,
<
http://www.princeton.edu/~harnad/psyc.html>), which entered
its second decade with the millennium.
Both journals are rigorously refereed; only those papers that
have successfully passed through the peer review filter go on
to run the gauntlet of open peer commentary, an extremely
powerful and important SUPPLEMENT to peer review, but certainly
no SUBSTITUTE for it. Indeed, no one but the editor sees [or
should have to see] the population of raw, unrefereed
submissions, consisting of some manuscripts that are eventually
destined to be revised and accepted after peer review, but also
(with a journal like BBS, having a 75% rejection rate) many
manuscripts not destined to appear in that particular journal
at all. Referee reports, some written for my eyes only, all
written for at most the author and fellow referees, are nothing
like public commentaries for the eyes of the entire learned
community, and vice versa. Nor do 75% of the submissions
justify soliciting public commentary, or at least not
commentary at the BBS level of the hierarchy.
It has been suggested that in fields such as Physics, where the
rejection rate is lower (perhaps in part because the authors
are more disciplined and realistic in their initial choice of
target journal, rather than trying their luck from the top
down), the difference between the unrefereed preprint
literature and the refereed reprint literature may not be that
great; hence one is fairly safe using the unrefereed drafts,
and perhaps the refereeing could be jettisoned altogether.
--------------------------------------------------------------------
Stevan Harnad harnad_at_cogsci.soton.ac.uk
Professor of Cognitive Science harnad_at_princeton.edu
Department of Electronics and phone: +44 23-80 592-582
Computer Science fax: +44 23-80 592-865
University of Southampton
http://www.ecs.soton.ac.uk/~harnad/
Highfield, Southampton
http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM
NOTE: A complete archive of this ongoing discussion of providing free
access to the refereed journal literature is available at the American
Scientist September Forum (98 & 99 & 00):
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html
You may join the list at the site above.
Discussion can be posted to:
american-scientist-open-access-forum_at_amsci.org
Received on Mon Jan 24 2000 - 19:17:43 GMT