Stevan Harnad writes:
> I am certain that online implementation will make (and already is
> making) CLASSICAL peer review faster, cheaper, more efficient, and more
> equitable. That can be confidently stated. But what (in my opinion) has
> to be avoided at all costs is any linking whatsoever between
> self-archiving (i.e., author/institution steps taken to maximize the
> visibility, accessibility, usage, citation and impact of their
> peer-reviewed research output) and any substantive changes in classical
> peer review.
That seems to me to be an untenable view. Any time a major change
takes place in method of dissemination of scholarly information,
changes in peer review are basically unavoidable. They may only
be changes that make what you call classical peer review better, but
that is a very unlikely course. It is much more probable that the
changes will be deeper.
> Classical peer review is merely the evaluation of the work of
> specialists by their qualified fellow-specialists (peers) mediated by
> and answerable to a designated qualified-specialist (the editor) who
> picks the referees, adjudicates the reports, indicates what needs to be
> done to revise for acceptance (if anything) and is answerable for the
> results of this quality-control, error-corrective mechanism.
>
> Untested "reforms" to this system, though possible, should not be
> mentioned at all, in the same breath as self-archiving, for any implied
> coupling between self-archiving and hypothetical peer-review changes
> will only work to the disadvantage of self-archiving and open access:
In other words, self-archiving is the preeminent goal, and we should
keep quite about any changes it might bring to peer review in order not
to frighten the uncommitted?
How does this differ from somebody a decade or two that might have
promised that electronic publishing would simply mean that journals
would now be available online, but there would be no disturbing
innovations such as scholars being confused by uncontrolled preprint
distribution?
>ao> This system is really a collection of many different systems,
>ao> of varying effectiveness. They guarantee neither correctness
>ao> nor novelty of the results, even among the most selective and
>ao> prestigious journals.
>
> No human (or nonhuman) judgement can guarantee that. The only relevant
> question -- and it has not been asked or tested, but the default
> assumption until it is tested MUST be for, not against, the causal role
> of peer review in maintaining the current quality level of the research
> literature -- is: How much better or worse is the literature's quality
> with (1) classical peer review, (2) with hypothetical (not yet tested
> and compared) alternatives, or (3) with no peer review at all (which,
> by the way, is NOT tested already by existing pre-refereeing preprint
> quality levels, for the invisible-hand reasons I've elaborated)?
And as electronic publishing became a possibility, would it not have
been natural to complain that the only way to maintain the quality of
scholarly publication was to insist on proven techniques (thus
ruling out self-archives, and extending the Ingelfinger rule to cover
all journals)?
> Absent the comparative data, there is only speculation (speculation that
> may well put the quality of the current refereed literature at risk if
> it were implemented before successful pre-testing). This is the sort
> of speculation from which I think it is so important to dissociate the
> question of self-archiving, completely. Any implied coupling will simply
> lose us yet another generation of potential self-archivers.
Again, by this line of reasoning, moving journals online should have been
carefully dissociated from irresponsible talk aout self-archiving and its
"pollution" of the literature.
> almost nowhere is peer-review merely red-light/green-light
> screening: Papers are not just refereed for acceptance/rejection.
> Referees propose corrections and elaborations, papers are revised and
> re-refereed. Peer-review is not a passive, static filter but an active,
> dynamic, interactive, corrective one.
So are (even to a greater degree) the many other stages of the
"scholarly skywriting" continuum.
> Without that dynamic, answerable, pre-correction, and without the
> tried-and-tested quality-label of an established journal to sign-post
> the skyline, I am convinced that the literature would not only quickly
> decline in quality, but it would become un-navigable -- till peer review
> was simply reinvented!
It is my contention that peer review is being reinvented, or more
precisely, reshaped. I do not deny the importance of review by peers,
but do question whether classical peer review is all that important.
It just has too many warts!
> Yet it is precisely this doomsday scenario that is holding would-be
> self-archivers back today, and I'm afraid you may just be reinforcing
> their fears here, Andrew!
But what I am holding out is the promise of an improved system of
review by peers.
> I sense (I am reading this sequentially in real time) that we are about to
> come to the "open peer commentary" alternative to "classical peer review":
> http://cogprints.soton.ac.uk/documents/disk0/00/00/16/94/index.html
You sense incorrectly. In the extremely short space I had, I could
not discuss open peer commentary in detail. It is likely to be an
element of future review systems, but I do not venture to predict
how important it will be.
> The self-correction in classical peer review is systematic, deliberate,
> and answerable (on a journal by journal basis). The ad-lib
> self-correctiveness of self-appointed sleuths tends more toward an
> opinion poll than expert guidance.
The "self-correction in classical peer review" is sadly inadequate.
I wrote at length about this in "Tragic loss or good riddance ...,"
and there are plenty of more systematic sources of complaints (for
example, the recent "publish and be damned ..." by David Adams and
Jonathan Knight in Nature, vol. 419, 24 Oct. 2002, pp. 772-776.
The supposedly gold standard of classical peer review is made of
badly corroded pewter! The recent Bell Labs scandal with Jan
Hendrik Schoen's fraudulent publications (many in Science and
Nature) is just the tip of the iceberg.
> In this new "system" we would be entrusting all of that to the four
> winds!
Hardly. We would be able to set our filters any way we wanted. We
could choose to look only at something that had been vetted by experts
of a top caliber (or, as an extreme example, only look at papers that
were at least 10 years old and had been mentioned favorably in half
a dozen survey articles in journals published by a given field's
main professional society), or we could accept all the recent postiging
to arXiv and other archives.
> Andrew, both of us are frustrated by the slowness with which the
> research community is coming to the realization that open access is the
> optimal and inevitable outcome for them, and that self-archiving is the
> way to get there. But do you really believe that inasmuch as they are
> being held back by fears about peer review this paper will embolden them,
> rather than confirming their worst fears?
I believe it is imperative to be honest. A move to self-archiving
will, I am convinced, lead to major changes in peer review, of the
type I am describing. Not right away, since time scales are
different, but eventually it will.
> Yet it is all completely unnecessary! All that's needed for open access
> is to self-archive, and leave classical peer review alone! Why imply
> otherwise?
Yes, and we could have promised scholars that electronics would
only lead to journals moving online, and that nobody would be
allowed to take advantage of the new freedoms to self-archive
their articles. That surely would have allayed the concerns
of many (especially of publishers).
> You are making predictions and conjectures, which is fine. But why link
> them to open-access and especially the current unfortunate reluctance to
> self-archive? Speculations will not relieve fears, especially not
> speculations that tend to confirm them.
I will deemphasize the link in my next revision, but will leave some
reside of it there. Anything else I feel would not be responsible.
> The law-review case, about which I have written and puzzled before,
> is an anomaly, and, as far as I know, there are many legal scholars
> who are not satisfied with it (Hibbitts included). (Not only are
> law-reviews student-run, but they are house organs, another anomaly in the
> journal-quality hierarchy, where house-journals tend to rank low, a kind
> of vanity-press.) I think it is highly inadvisable to try to generalize
> this case in any way, when it is itself unique and poorly understood. In
> any case, it certainly will not be reassuring to professors who are
> contemplating whether or not they should self-archive, that doing so
> may mean that whereas they are marking their students essays on
> tuesdays and thursdays, if they self-archive their own papers, their
> students may be marking them on wednesdays and fridays, instead of the
> qualified editor-mediated peers of times past.
The law review case may be "poorly understood," but so is the whole
classical peer review system. It does, however, serve as a counterexample
to many extreme claims about what kind of review is needed. That many
scholars are not satisfied with it is nothign special. The same can
be said of classical peer review.
>ao> The growing role of interdisciplinary
>ao> research might lead to a generally greater role for non-peers in reviewing
>ao> publications.
>
> I can't follow this at all. Interdisciplinary work requires review by
> peers from more disciplines, not from non-peers. ("Peer" means qualified
> expert.)
If I, as a mathematician, need to rely on some results from physics,
I may end up criticizing the presentation and methodology of a
physics paper even without understanding all the physics that is
involved.
It is a weak analogy I would not want to push too far, but note that
many music teachers and sports coaches are very successful, and
train top stars in their areas, without being able to perform at
their students' level.
>ao> However, in most cases only peers are truly qualified to review
>ao> technical results. However, peer evaluations can be obtained, and
>ao> increasingly are being obtained, much more flexibly than through
>ao> the traditional anonymous journal refereeing process.
>
> That is not my experience. It seems that qualified referees, an
> overharvested resource, are becoming harder and harder to come by. They
> are overloaded, and take a long time to deliver their reports. Is the
> idea that they will be more available if approached some other way? Or
> if they self-select? But what if they all want to review paper X, and no
> one -- or dilettantes -- review papers A-J?
You help make my case. Classical peer review typically is too slow,
and it is getting harder to run. Self-selection is a major antidote.
Yes, it is not ideal, as indeed, interests of potential referees
won't be uniformly distributed, but I will settle for that if I can't
get anything better. As the primality example later on shows, it
is the most important articles that are likely to get the fastest
and most thorough scrutiny, nad that is as it should be.
>ao> Some can come from use of automated tools to harvest references
>ao> to papers, in a much more flexible and comprehensive way than the
>ao> Science Citation Index provided in the old days.
>
> Now here I agree, but this falls squarely in the category of using
> online resources to implement CLASSICAL peer review more efficiently and
> equitably: Here, it is to help find qualified referees and to distribute
> the load more evenly. But that has nothing to do with peer review
> reform, nor with any of the other speculative alternatives considered
> here. It goes without saying that an open-access corpus will make it
> much easier and more effective to find qualified referees.
Again, I have a different view. If I am looking for something in
psychology, and area I know very little about, and find a relatively
recent archives paper that has not been published, but is referenced
favorably by Stevan Harnad and several other famous figures, should
I not be willing to accept it as of good quality?
> There's another way to put all this: To a first approximation (and
> forgetting about what I said about dynamic correction, revision etc.),
> a journal's quality level is a function of its rejection rate: The
> highest quality journals will only accept the highest quality work,
> rejecting the rest. Second-tier journals will reject less, and so on,
> right down to the near-vanity press at the bottom, which accepts just
> about anything. This is the hierarchy of sign-posted milestones that
> guides the prospective reader and user rationing his finite reading time
> and his precious research resources. How is this quality triage to be done
> on the model you just described (of the prime-number flurry)?
I would dispute the claimed strong correlation between rejection
rates and quality. Having served on the editorial board of what
is usually regarded as one of the three most prestigious journals
in mathematics, I can say that its rejection rate was actually
lower than of several lower quality journals I have served on.
The reason was self-selection. Aside from a moderate fraction
of crank submissions (something like 10 to 20%), the overwhelming
majority were of very high quality. Authors knew of the journal
standards, and did not bother to submit run-of-the mill papers.
This is just one anectodal piece of evidence, but from what
I have heard from other editors, is not all that atypical.
> Andrew, I'm curious: experiences as what in those areas: reader? author?
> referee? editor? empirical investigator of peer-review?
All of the above.
>ao> I believe that the frequently voiced concerns about need for extra
>ao> scrutiny of research results that might affect health practices
>ao> are a red herring. Yes, decision about medical procedures or even
>ao> diet should be based on solidly established research. However,
>ao> the extra levels of scrutiny are more likely to be obtained by more
>ao> open communication and review systems than we have today.
>
> And a little bit of self-poisoning by the users after the
> self-publicizing by the authors, by way of self-correction?
Well, it seems users are able to self-poison themselves quite well
as is. Didn't the President of South Africa discover on the Internet
that AIDS is not caused by a virus?
> Andrew, I'm afraid I disagree rather profoundly with the position you
> are advocating here! I think it is far more anecdotal and speculative
> than your other work on publishing and access. I think the conjectures
> about peer review are wrong, but worse, I think they will be damaging
> rather than helpful to self-archiving and open access. I think my own
> article, which you cite, actually premptively considered most of these
> points already, more or less along the lines I have repeated in
> commentary here. I have to say that I rather wish that you weren't
> publishing this -- or at least that you would clearly dissociate it from
> self-archiving, and simply portray it as the conjectures they are, from
> someone who is not actually doing research on the peer review system,
> but mere contemplating hypothetical possibilities.
What I am saying here is pretty much what I have been saying ever
since "Tragic loss or good riddance ..." back in 1994. My position
has not changed. This is an opinion piece, as solicited by the
editors of this volume, so it certainly is largely personal
evaluation and speculation. However, everything I have seen in
the last 8 years confirms my initial impression.
> Or failing that, I wish I could at least write a commentary by way of
> rebuttal!
Why don't you propose it to the editors? (BTW, mine is just one of
several short contributions they have solicited. I have not seen
any of the others.)
Andrew Odlyzko
Received on Tue Nov 05 2002 - 21:56:50 GMT