On Wed, 6 Nov 2002, Andrew Odlyzko wrote:
>sh> (R1) Your primary motivation is (R) to reform research
>sh> communication, mainly through the posting of all papers online,
>sh> unrefereed, and then relying on self-selected "vetters," in
>sh> place of of classical peer review, to evaluate, improve and
>sh> thereby sign-post paper quality.
>ao> I would not phrase it this way. I would say that my primary motivation
>ao> is indeed to reform research communication, mainly through the posting
>ao> of all papers online, unrefereed, and then relying on whatever mixture
>ao> of classical peer review, or contributions of self-selected "vetters,"
>ao> the community decides to rely on. I do not hold dogmatic views of how
>ao> peer review will be handled, and predict only rough trends.
It is gratifying to know that you are only predicting rough trends.
It is important to make it quite explicit here just how close our
position is, so as to pinpoint exactly what it is that we disagree on
(i.e., the trends you predict):
(1) My motivation is freeing access to all research, both pre- and post
peer review (i.e., nut just unrefereed preprints), through self-archiving.
(2) If we could (for some arbitrary reason!) free only one of the two, I
would choose the refereed final draft rather than the unrevised preprint,
but there is no reason not to free both (with a few special exceptions:
see below).
(3) Given success in convincing the world research community to free
access to their research by self-archiving both their preprints and
their postprints, we would eo ipso also have all the benefits of
"self-selected vetting" that you (and I!) both value. But this would be
a *supplement* not a *substitute* for classical peer review. (Nothing
whatever would be lost, and everything would be gained!)
(4) If I were you, and I believed self-selected vetting will eventually
replace classical peer review, I would not feel any need to add anything
to (1) - (3). Self-archiving, and the open access it brings, would be
all I would need to evangelize for. The rest (if my belief that vetting
could and would replace peer review was correct) would then take care
of itself.
(5) But self-archiving is (we both agree) taking place far too slowly.
(6) Among the reasons self-archiving is taking place so slowly are
worries that researchers have that hold them back from self-archiving.
The two main ones are:
(i) the worry that self-archiving would compromise or destroy peer
review
http://www.eprints.org/self-faq/#7.Peer
and
(ii) the worry that self-archiving would violate copyright
http://www.eprints.org/self-faq/#10.Copyright
(7) As we both regard open access as optimal, inevitable, and long
overdue, we should both be concerned with relieving worries (i) and
(ii).
(8) Your belief that self-vetting will eventually replace classical
peer review is one that would *reinforce* rather than relieving
researchers' worry on that score. Hence, unless you could persuade them
not only that it will happen, but that it too is optimal, voicing your
belief (and it is only a belief, a hypothesis), even as the prediction
of a "rough trend" seems more likely to slow rather than speed the
transition to what we both agree is the optimal and the inevitable outcome
(open access).
(9) I believe strongly that your belief (that self-vetting will replace
classical peer review) is wrong, and that there is no trend, rough
or smooth, in that direction, now till there be. (I believe that, not
dogmatically, but for the very concrete reasons I have repeatedly adduced
in this series of exchanges.) But if someone like me -- who believes fully
in self-archiving and the transition to open-access, and disbelieves in
any causal connection between that and any risk to classical peer
review -- is not persuaded by your contrary belief, nor the arguments
you adduce in its support, then how likely is it that someone who does
not yet believe in self-archiving *and* worries that it would be a risk
to peer review will be emboldened (to self-archive) by your hypothesis
(even formulated as a "rough trend")?
(10) The optimality of open access to the research literature is a
certainty, not a hypothesis. We both agree about that, and about the ample
evidence that it maximizes research visibility, accessibility, uptake,
usage, citation, and impact, as well as scope, speed, and interactivity,
in short, that it greatly benefits research and researcher productivity.
(11) The transition from classical peer review to self-selected vetting,
in contrast, is merely a hypothesis (and one with -- in my view -- much
a priori evidence and many reasons to conclude that it is incorrect, but
never mind) -- a hypothesis that, on the face of it, looks as if it
would retard the transition to open access through self-archiving by
reinforcing the worries of those who do not self-archive precisely
because they are afraid it would destroy classical peer review!
(11) So why even voice the hypothesis? For if it is true, then it
will be a causal consequence of self-archiving anyway; hence, since
self-archiving is the necessary and sufficient condition for it,
everything should be done to hasten self-archiving. Yet voicing this
(fallible) hypothesis is very likely to slow, rather than hasten
self-archiving.
(12) As to my (arbitrary, desert-island) forced choice of open access
for the peer-reviewed final draft over the unrefereed preprint, not only
is it not a choice anyone has to make (since there is in most cases no
reason not to have both), but I even have an explicit commitment to the
self-archiving of unrefereed preprints, for two reasons:
(13) The first benefit of self-archiving the unrefereed preprint is one
on which we both agree -- that it hastens and strengthens the progress,
interactivity and self-correctiveness of research.
(14) The second benefit of self-archiving the unrefereed preprint is
that it is a means of legally self-archiving even in the face of the
most restrictive copyright transfer agreement: The self-archiving of the
preprint pre-dates any agreement or even any submission to a journal;
and after submission, refereeing, revision, and acceptance, if the
copyright agreement forbids self-archiving the final draft, one need only
self-archive the corrections that would have to be made by the user on
the already public preprint to make it conform to the final draft.
http://www.eprints.org/self-faq/#publisher-forbids
(15) Hence, besides being highly beneficial in its own right, the
self-archiving of the unrefereed preprint is an ally in ensuring open
access to the contents (if not the form) of the refereed final draft too.
(16) So we agree on the importance and sure benefits of self-archiving
preprints and we agree on the importance and sure benefits of open
access. We disagree only on your hypothesis that self-archiving will
eventually lead to the replacement of classical peer review by
self-selected vetting (or that there is any "rough trend" in that
direcrion.)
(17) Yet we agree that even if the hypothesis is correct, all it needs
is self-archiving -- and we both believe fully in the importance and
benefits of self-archiving.
(18) Perhaps what self-archiving needs now is facts rather than
hypotheses. The benefits of self-archiving are demonstrated facts,
whereas your hypothesis remains completely untested (and the likelihood
of success for any eventual experiment would seem to be contradicted by
a number of prima facie logical and practical considerations that I have
repeatedly listed in these exchanges).
(19) There is only one remaining problem with emphasizing (as you do),
only the self-archiving of pre-refereeing preprints, rather than also
the refereed postprints:
(20) There are special cases -- I couldn't say how many, or how their
distribution varies by field, but certainly a non-zero number, as the
"preprint culture" of physics is by no means universal yet -- in which
the author would prefer not to make his research public before it is
(classically) peer-reviewed. (There are also cases, particularly in
the medical literature, where the public posting of unrefereed findings
might represent a danger to public health.)
http://www.nih.gov/about/director/ebiomed/com0509.htm#harn45
(21) This is a further reason for emphasizing the self-archiving of the
refereed postprint. Otherwise, such authors/papers are also lost to
self-archiving altogether.
(22) If such authors go on to sign over-restrictive copyright transfer
agreements after refereeing then they may still be lost to self-archiving
for the time-being, because for them the preprint-plus-corrigenda strategy
above (14) will not work (there being no already-archived preprint);
but there seems no reason to lose open access to the research of all
preprint non-archivers by restricting self-archiving to preprints alone,
rather than preprints and postprints.
(23) Finally, many sceptics about the benefits of open access will only
be won over once the refereed, published postprints are openly
accessible, and not merely the unrefereed preprints.
(24) Having said all this, I have sufficient confidence in the
self-correctiveness of open online communication (such as this exchange)
not to worry too much about the possible untoward effects of the airing
of your hypothesis on those who are reluctant to self-archive because
of worries about peer review. As long as both sides are aired, let us
trust the outcome to (self-corrective) human judgment. (I hope Fiona
Godlee will agree to co-publish my reply along with your hypothesis in
the collection in which it will appear. Thanks for agreeing to post this
exchange to the American Scientist Forum.)
>sh> (A1) My primary motivation is (A) to make the classically peer-reviewed
>sh> research literature we have now openly accessible online, such as
>sh> it is, to maximize its accessibility, usage and impact. Qualified
>sh> journal editors continue to select the referees, referees and author
>sh> remain answerable to the editor, and the journal-name and its quality
>sh> track-record sign-post for potential users that an article has met
>sh> that journal's quality-standards.
>sh>
>sh> (R2) You see a causal connection between A and R: A will lead to R.
>
>ao> Yes, provided that (A) involves open access to preprints. (The way you
>ao> have phrased (A), it could encompass a system in which the Ingelfinger
>ao> rule would be universal, it's just that after publication by a journal,
>ao> articles would be freely available. That form of (A) would not yield
>ao> to (R).)
I agree, and I hope that what I wrote above now shows that this is not
at all what I meant. I am both strongly in favor of the self-archiving of
preprints and strongly opposed to the Ingelfinger rule and have published
critiques of it. However, I am at least equally in favor of the
self-archiving of the refereed postprints too:
Harnad, S. (2000) Ingelfinger Over-Ruled: The Role
of the Web in the Future of Refereed Medical Journal
Publishing. Lancet Perspectives 256 (December Supplement): s16.
http://cogprints.soton.ac.uk/documents/disk0/00/00/17/03/
Harnad, S. (2000) E-Knowledge: Freeing the Refereed
Journal Corpus Online. Computer Law & Security Report
16(2) 78-87. [Rebuttal to Bloom Editorial in Science
and Relman Editorial in New England Journal of Medicine]
http://cogprints.soton.ac.uk/documents/disk0/00/00/17/01/
>sh> (R4) You think that the research advances that occur before peer review
>sh> through the online posting of pre-refereeing preprints today are evidence
>sh> that peer review is unnecessary and can be replaced by spontaneous vetting
>sh> (R) without loss (and perhaps even with a gain) in quality.
>
>ao> Not that "peer review is unnecessary," but that "classical peer review is
>ao> unnecessary." I would also quarrel with the phrasing of the last part
>ao> of this point, but will let it go for lack of time.
It would be useful to know precisely what you mean -- especially since
this exchange is really meant to clarify things for those who may be
worried that self-archiving poses a risk to classical peer review. I am
convinced that there is no causal connection whatsoever between
self-archiving and changes in classical peer review (except that online
pre-refereeing feedback is a useful supplement to peer review, and that
the online medium will make it possible to implement classical peer
review more quickly, cheaply, efficiently and equitably).
Hence there is no reason to refrain from self-archiving because of
worries about peer review:
http://www.eprints.org/self-faq/#7.Peer
On the other hand, a system consisting exclusively of self-selected
online vetting is no form of peer review, classical or otherwise (except
in those cases where it happens to happen by chance!).
>sh> There is not only this total empirical gap between the data you
>sh> use and the conclusions you draw, but there are also logical
>sh> gaps: You have not replied when I have asked how, in a system
>sh> where classical peer review and journal-names with track-records
>sh> are no longer there as the back-up and bottom line -- as they are
>sh> universally and without exception now -- how the annual 2,000,000
>sh> papers (which are today refereed and sign-posted by refereed
>sh> journals) will find their proper vetting, and how this will be
>sh> sign-posted for potential users? This question does not even come
>sh> up in the case of pre-refereeing preprints, because those are a
>sh> "parallel economy," backed up by the classical peer-review and
>sh> then sign-posted by the names and track-records of the journals to
>sh> which just about every one of those preprints has been submitted,
>sh> and in which they will all appear eventually (though perhaps only
>sh> after several rounds of revision and refereeing, and perhaps not
>sh> always at the level of the journal to which they were submitted
>sh> first.)
>
>ao> I did not reply because I did not have time to reply to all of your
>ao> points. Since you make this such a central point, let me respond
>ao> now (although very briefly and so inadequately). How will all
>ao> those papers "find their proper vetting"? Well, how do they find
>ao> proper refereeing now, under your vaunted classical peer review?
By being submitted to a journal, whose editor (presumably a recognized
expert in the field) is responsible for (1) selecting qualified referees
(busy, reluctant, but willing-if-asked by the right journal/editor),
(2) deciding which of the referees' recommendations are valid and need
to be satisfied in order to meet the journal's established quality
standards, and (3) making sure the accepted, final draft has satisfied
them. The result is then recognizably (4) tagged (sign-posted) as
having been thus quality-controlled by the journal-name (and associated
track-record). That's all there is to classical peer review: Human expert
judgment, systematized, answerable, and reliably labelled accordingly.
How does anarchic self-selected vetting ensure an equivalent outcome,
and how is it to be sign-posted?
>ao> We know that serious frauds like the Jan Hendrik Schoen slip through.
I have replied about fraud in earlier postings. In brief, fraud will
be found out anyway, because one cannot build on it, and research
progress is about building cumulatively on findings, not merely reporting
them. But, in any case, with open-access preprints and postprints as
a supplement to peer review, all the benefits of the extra lines of
defence are there, over and above peer review, without any need to change
classical peer review in any way (other than to make it faster and
more efficient in finding and reaching referees and in distributing the
refereeing load more broadly and evenly).
Re: A Note of Caution About "Reforming the System"
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2322.html
>ao> We also have plenty of evidence that lots of simply
>ao> not very solid science that is not fraudulent gets through.
Yes, human judgment, even expert peer judgment, is fallible. And
supplementing its systematic, answerable, labelled application with
open feedback will be very useful: But why construe this supplement
as a substitute? Why would it not simply co-exist with the classical
line of defense?
>ao> (I don't have time to dig up references, but there was a paper
>ao> quite a while ago that looked at the statistical methodology used
>ao> in a large sample of medical papers. It found a horrendously high
>ao> rate of misapplications of statistics.
Horrendous compared to what? Compared to how many would have been found
if the raw preprints had all simply been publicly posted, hoping that
the right vetters would find and test them all for us all spontaneously?
What evidence is there that they could or would, without the systematic
mediation of someone who was qualified and responsible for the outcome
(such as a journal editor)?
Or would attention have to be drawn by someone having poisoned
himself on the basis of an unrefereed remedy first?
http://www.nih.gov/about/director/ebiomed/com0509.htm#harn45
On the other hand, to repeat, open feedback on an open access classically
refereed literature, pre-, during, and post- refereeing, would certainly
be a welcome second line of defence, and would no doubt help improve
the quality of the research literature.
>ao> There are lots more examples.) The point is that classical peer
>ao> review does not provide much of a signal, especially for journals
>ao> in the lower quality tiers.
And you think self-selected feedback would provide at least as much of a
"signal," especially for journals in the lower quality tiers?
(Sometimes I think what you are saying is that the elite work does not
really need peer review and for the rest it doesn't really matter...!)
>ao> So how does science progress? Well, there are all sorts of checks
>ao> that are applied post-publication. (And none of them are infallible.
>ao> Even a few Nobel prizes are now regarded as having been given in error.)
>ao> Basically classical peer review is just one noisy and uncertain signal
>ao> that the scholarly community relies on.
Classical peer review is not a signal; it is a dynamic, interactive
quality-control and tagging system -- and the only one that is systematic
and answerable. I cannot see any way that anarchic self-selected feedback
can replace this (other than by re-inventing classical peer review under
another name) -- though I can see how it can (and does) complement it.
>sh> If self-archiving had (mirabile dictu) begun instead with refereed
>sh> postprints [rather than unrefereed preprints], we might have
>sh> spared ourselves these misconstruals [about peer review] and
>sh> we might have been further along the road to open access by now...
>
>ao> The incentives were not there to do this. The authors, who after all
>ao> control the information flow, could see the benefits to themselves
>ao> of quick circulation of preprints. Open access to published journal
>ao> articles was of much less value to them, since they typically had access
>ao> to those journals in their libraries.
I agree that those were the initial conditions that turned it into a
historical fact that self-archiving en masse began with physicists,
self-archiving their pre-refereeing preprints first (though soon also
their published postprints too). They were already a (paper) "preprint
culture" and (the elite among them) did not lack access to the
toll-based journal literature.
That's why I said it would have been "mirabile dictu" if it had begun
instead with refereed postprints. But that is all history now, and our
eyes are opened, and the data are in, and it should now be clear that open
access to the entire research literature, before and after refereeing,
is what will be optimal for all. And that what peer-reviewed research
needs now is to be freed from access-blocking tolls, not from
quality-controlling peer review.
Stevan Harnad
Received on Thu Nov 07 2002 - 17:01:57 GMT