A few more responses, again very brief due to lack of time.
P.S. An interesting source for data and references about peer
review (including costs) that I have just learned about is the
recent paper of Fytton Rowland, "The peer-review process,"
Learned Publishing, vol. 15, no. 4, Oct. 2002, pp. 247-258,
available online at
<
http://rudolfo.ingentaselect.com/alpsp/09531513/v15n4/>.
On Thu, 7 Nov 2002 Stevan Harnad <harnad_at_ecs.soton.ac.uk> wrote:
> It is important to make it quite explicit here just how close our
> position is, so as to pinpoint exactly what it is that we disagree on
> (i.e., the trends you predict):
> (1) My motivation is freeing access to all research, both pre- and post
> peer review (i.e., nut just unrefereed preprints), through self-archiving.
> (2) If we could (for some arbitrary reason!) free only one of the two, I
> would choose the refereed final draft rather than the unrevised preprint,
> but there is no reason not to free both (with a few special exceptions:
> see below).
Here we differ, in that I would opt for freeing the unrevised preprint,
if for no other reason that it is much easier. (In general, my papers
are more descriptive, writing about what is happening, and what is likely
to happen, and less prescriptive, writing about what I would like to see
happen. Yours are the other way.)
> (3) Given success in convincing the world research community to free
> access to their research by self-archiving both their preprints and
> their postprints, we would eo ipso also have all the benefits of
> "self-selected vetting" that you (and I!) both value. But this would be
> a *supplement* not a *substitute* for classical peer review. (Nothing
> whatever would be lost, and everything would be gained!)
Absolutely. As I have been emphasizing, there would be nothing to stop
the scholarly community from continuing with classical peer review the
way they have been doing. It's just that I don't believe it would happen
that way. Here is yet another imperfect analogy (imperfect because I don't
have time to think of a better one): When radio (and then TV) came in,
one could have argued that the world could have continued to rely on
newspapers for news just as much as before, with radio and TV just
supplements. All that was happening was that more options were being
offered. However, once the radio and TV options did become available,
they were embraced enthusiastically, and although newspapers continue,
their roles in information dissemination have changed.
> (4) If I were you, and I believed self-selected vetting will eventually
> replace classical peer review, I would not feel any need to add anything
> to (1) - (3). Self-archiving, and the open access it brings, would be
> all I would need to evangelize for. The rest (if my belief that vetting
> could and would replace peer review was correct) would then take care
> of itself.
Agreed.
> (5) But self-archiving is (we both agree) taking place far too slowly.
Agreed, again.
> (6) Among the reasons self-archiving is taking place so slowly are
> worries that researchers have that hold them back from self-archiving.
> The two main ones are:
> (i) the worry that self-archiving would compromise or destroy peer
> review http://www.eprints.org/self-faq/#7.Peer
> and
> (ii) the worry that self-archiving would violate copyright
> http://www.eprints.org/self-faq/#10.Copyright
Yes again.
> (7) As we both regard open access as optimal, inevitable, and long
> overdue, we should both be concerned with relieving worries (i) and
> (ii).
Agreed.
> (8) Your belief that self-vetting will eventually replace classical
> peer review is one that would *reinforce* rather than relieving
> researchers' worry on that score. Hence, unless you could persuade them
> not only that it will happen, but that it too is optimal, voicing your
> belief (and it is only a belief, a hypothesis), even as the prediction
> of a "rough trend" seems more likely to slow rather than speed the
> transition to what we both agree is the optimal and the inevitable outcome
> (open access).
I do not necessarily accept the first sentence.
> (9) I believe strongly that your belief (that self-vetting will replace
> classical peer review) is wrong, and that there is no trend, rough
> or smooth, in that direction, now till there be. (I believe that, not
> dogmatically, but for the very concrete reasons I have repeatedly adduced
> in this series of exchanges.) But if someone like me -- who believes fully
> in self-archiving and the transition to open-access, and disbelieves in
> any causal connection between that and any risk to classical peer
> review -- is not persuaded by your contrary belief, nor the arguments
> you adduce in its support, then how likely is it that someone who does
> not yet believe in self-archiving *and* worries that it would be a risk
> to peer review will be emboldened (to self-archive) by your hypothesis
> (even formulated as a "rough trend")?
The logic here is deficient. You are assuming that those skeptical of
self-archiving are enthusiastic about classical peer review. Yet if we
look at the literature on classical peer review, we see a lot of concern
about its deficiency, dating back many decades, before self-archiving was
even a possibility.
> (10) The optimality of open access to the research literature is a
> certainty, not a hypothesis. We both agree about that, and about the ample
> evidence that it maximizes research visibility, accessibility, uptake,
> usage, citation, and impact, as well as scope, speed, and interactivity,
> in short, that it greatly benefits research and researcher productivity.
We both agree on this optimality, but that is not a universal opinion.
I think there is still considerable concern, especially in the biomed
community, of letting the laity have access even to the peer-reviewed
literature.
> (11) The transition from classical peer review to self-selected vetting,
> in contrast, is merely a hypothesis (and one with -- in my view -- much
> a priori evidence and many reasons to conclude that it is incorrect, but
> never mind) -- a hypothesis that, on the face of it, looks as if it
> would retard the transition to open access through self-archiving by
> reinforcing the worries of those who do not self-archive precisely
> because they are afraid it would destroy classical peer review!
But it is not a hypothesis. For leading researchers in the areas that
have embraced arXiv, say, it is a reality, one that works. Many other
examples are presented in my "The rapid evolution of scholarly communication."
> (11) So why even voice the hypothesis? For if it is true, then it
> will be a causal consequence of self-archiving anyway; hence, since
> self-archiving is the necessary and sufficient condition for it,
> everything should be done to hasten self-archiving. Yet voicing this
> (fallible) hypothesis is very likely to slow, rather than hasten
> self-archiving.
Even if this was just a hypothesis, I think basic intellectual standards
require us to be open about the likely consequences of self-archiving.
> (16) So we agree on the importance and sure benefits of self-archiving
> preprints and we agree on the importance and sure benefits of open
> access. We disagree only on your hypothesis that self-archiving will
> eventually lead to the replacement of classical peer review by
> self-selected vetting (or that there is any "rough trend" in that
> direcrion.)
Yes, except I feel my predictions are more than just a hypothesis.
> (17) Yet we agree that even if the hypothesis is correct, all it needs
> is self-archiving -- and we both believe fully in the importance and
> benefits of self-archiving.
Yes, after many years of being active in this area, I do not have any
illusions that my writings are going to have a big impact on scholarly
publishing.
> (18) Perhaps what self-archiving needs now is facts rather than
> hypotheses. The benefits of self-archiving are demonstrated facts,
> whereas your hypothesis remains completely untested (and the likelihood
> of success for any eventual experiment would seem to be contradicted by
> a number of prima facie logical and practical considerations that I have
> repeatedly listed in these exchanges).
Again, if you want a test, just look at what happens in arXiv.
> How does anarchic self-selected vetting ensure an equivalent outcome,
> and how is it to be sign-posted?
I skipped through some earlier points, but let me just say a few words
here that will address them as well as this one.
To take a simple example, the quality signal associate with a journal-name
is something rather nebulous. After all, it is not all that hard to start
up a new journal, and the perceived quality range of different journals is
really vast. Further, even a single journal contains papers of varying
quality levels. So how does a scholar use the signal that publication
in a journal conveys? Well, it is a very vague quality signal. In the
Gutenberg era, that was about all that was possible to obtain. Today,
though, there are a variety of other signals that people can collect nad
easily sift through (the subject of "The rapid evolution of scholarly
communication," <
http://www.catchword.com/alpsp/09531513/v15n1/contp1-1.htm>).
> Yes, human judgment, even expert peer judgment, is fallible. And
> supplementing its systematic, answerable, labelled application with
> open feedback will be very useful: But why construe this supplement
> as a substitute? Why would it not simply co-exist with the classical
> line of defense?
They coexist now, and will continue to do so for a long time. (As I wrote
in "The slow evolution of electronic publishing," sociological changes are
very slow.) The question is, which is going to be more important? That is
where we differ.
> >ao> There are lots more examples.) The point is that classical peer
> >ao> review does not provide much of a signal, especially for journals
> >ao> in the lower quality tiers.
> And you think self-selected feedback would provide at least as much of a
> "signal," especially for journals in the lower quality tiers?
Absolutely. Instead of a vague signal that some referee(s) decided the
submission was worth publishing (without explaining to the readers the
reasons for this judgement, of the quality evaluation), we could have
a much richer set of signals.
> (Sometimes I think what you are saying is that the elite work does not
> really need peer review and for the rest it doesn't really matter...!)
I believe that all work benefits from peer review, but the potentially
important work needs more of it, and will get more of it.
> >ao> So how does science progress? Well, there are all sorts of checks
> >ao> that are applied post-publication. (And none of them are infallible.
> >ao> Even a few Nobel prizes are now regarded as having been given in error.)
> >ao> Basically classical peer review is just one noisy and uncertain signal
> >ao> that the scholarly community relies on.
> Classical peer review is not a signal; it is a dynamic, interactive
> quality-control and tagging system -- and the only one that is systematic
> and answerable. I cannot see any way that anarchic self-selected feedback
> can replace this (other than by re-inventing classical peer review under
> another name) -- though I can see how it can (and does) complement it.
We'll just have to continue to disagree on this.
Andrew Odlyzko
Received on Sun Nov 10 2002 - 16:28:33 GMT