Dear Andrew, I think our exchange has been useful, if only to highlight
the points on which we will have to agree to disagree, and why. I will
try to summarize our four main points of disagreement, and then make
some comments on your recent replies. I will send our exchange to Fiona,
offering to cobble mine into a commentary (and suggesting that if she is
interested, she then ask you to do the same with your replies). I look
forward to breakfasting with you (and maybe Jean-Claude) on November 22.
One last question: As our differences are, I think, rather important,
because we are both very involved in the open-access movement and we
have both written a good deal about it, would you give me permission to
also post our full exchange in the American Scientist Forum? The BOAI
list on which they have appeared is not a public list, and seen only by
the dozen or so original signatories of the BOAI.
Summary of our differences:
(R1) Your primary motivation is (R) to reform research communication,
mainly through the posting of all papers online, unrefereed, and then
relying on self-selected "vetters," in place of of classical peer review,
to evaluate, improve and thereby sign-post paper quality.
(A1) My primary motivation is (A) to make the classically peer-reviewed
research literature we have now openly accessible online, such as
it is, to maximize its accessibility, usage and impact. Qualified
journal editors continue to select the referees, referees and author
remain answerable to the editor, and the journal-name and its quality
track-record sign-post for potential users that an article has met
that journal's quality-standards.
(R2) You see a causal connection between A and R: A will lead to R.
(A2) I see a causal connection between worries about R and not-A:
Worries that R would compromise or destroy the quality and usability
of the peer-reviewed literature hold researchers back from doing A.
(R3) You think peer review is flawed and should be replaced by something
better.
(A3) I think peer review is no more nor less flawed than any other
area of human judgment; it is merely a systematic, answerable method
for soliciting qualified judgment, and sign-posting the outcome. If
there is a better method, then it should definitely replace classical
peer review; but no one has yet tested or demonstrated a better
method.
(R4) You think that the research advances that occur before peer review
through the online posting of pre-refereeing preprints today are evidence
that peer review is unnecessary and can be replaced by spontaneous vetting
(R) without loss (and perhaps even with a gain) in quality.
(A4) I think those research advances are only evidence that online
pre-refereeing preprint posting is a very valuable supplement to --
but not a substitute for -- classical peer review, which is still
there, unchanged, alongside all preprint posting and exchange today
(the "invisible hand" of answerability to classical peer review).
No test of what would happen without classical peer review has
been done; all your evidence is parasitic on intact peer review.
On Tue, 5 Nov 2002, Andrew Odlyzko wrote:
> We do have substantially different visions of the future of peer review,
> and they have not changed much since we first started corresponding back
> in 1993.
I agree. Nor has the evidence changed since 1993.
>sh> at I am suggesting is that if your prediction [that A will lead to R,
>sh> and that R is viable] happens to be wrong, making the prediction
>sh> anyway will have a negative, retardant effect on self-archiving and open
>sh> access. (Of course, if you are right, then concerns about these changes
>sh> will still have a negative, retardant effect on self-archiving.)
>
> I am not sure it will be negative. It might encourage the transformation
> by showing the path to a future in which not only information dissemination,
> but also peer review, are improved.
If you are right (and if reluctant self-archivers, worried about peer
review, believe you). But if you are not right, or not believed, then
the effect will be negative.
> I hope to hasten the transition [to open access] by pointing out that
> it is likely to improve the peer review system.
And I hope that voicing your conjecture will not have the
opposite effect.
> Sorry, but alternatives to classical peer review have been tried, and
> are constantly being tried (quite successfully, in my opinion). That
> is largely what my article "The fast evolution of scholarly communication"
> was about.
Testing your hypothesis means performing systematic, controlled
experiments on representative and sufficiently large samples of the
literature (on a sufficient time-scale) to test whether a system
consisting exclusively of self-selected vetting of completely unrefereed
papers, with no subsequent or parallel classical peer review, yields a
literature of quality comparable to its current quality. Nothing faintly
like this has been attempted yet. Online preprint posting and exchange
occur within a "parallel economy," with classical peer review still 100%
in place and active, before, during and afterward.
There is not only this total empirical gap between the data you use and
the conclusions you draw, but there are also logical gaps: You have not
replied when I have asked how, in a system where classical peer review
and journal-names with track-records are no longer there as the back-up
and bottom line -- as they are universally and without exception now --
how the annual 2,000,000 papers (which are today refereed and sign-posted
by refereed journals) will find their proper vetting, and how this will
be sign-posted for potential users? This question does not even come up
in the case of pre-refereeing preprints, because those are a "parallel
economy," backed up by the classical peer-review and then sign-posted by
the names and track-records of the journals to which just about every
one of those preprints has been submitted, and in which they will all
appear eventually (though perhaps only after several rounds of revision
and refereeing, and perhaps not always at the level of the journal to
which they were submitted first.)
How will the 2,000,000 annual papers (all? some? most? enough? which?)
find their qualified vetters? And how will potential vetters know what
to vet (and why will they want to do that faithful duty for us all under
those ad lib conditions, busy as they all are)? And what will ensure that
the papers' authors (all? some? most? enough? which?) hear and heed what
needs to be heard and heeded? And how will potential users know whether
(and when) the vetting has been heeded? You, Andrew, are imagining that,
persisting miraculously under these anarchic conditions, there will still
be something much like what we have already, with classical peer review,
whereas I am imagining the un-navigable chaos it would quickly decline
into, once no longer propped up by the invisible hand of peer review
and the sign-posting of the outcome by the journals that implement it.
One can, of course, dream up a system of answerability and classification
that would systematize and bring order into all this. But call it what
you like, it will prove to be a re-invention of something very much like
classical peer review.
> Well, yes, we don't know whether my prediction is true, but there is
> evidence for it, for example in "The fast evolution ...." Peer review
> is undergoing change right now, in front of our eyes, even though
> few are paying attention. To pretend that nothing will change seems
> really short-sighted.
I submit that nothing like the test I described above is happening now.
It is all just small local supplements to the universal system currently
backing it up.
> We get a very weak quality and usability signal from
> classical peer review, and my contention is that we can obtain many other
> signals that collectively, if not individually, can be even more useful.
Classical peer review is, as I have agreed repeatedly, fallible, being
based on human judgment. Perhaps it can be improved; it can certainly be
supplemented. But whether an alternative is truly a substitute -- and
whether it can yield a literature of at least equal quality and usability
-- must first be tested. None of the data from which you draw your
conclusions constitute such a test; they are all parasitic on classical
peer review.
> I would have to go back and dig up some old messages, but I believe there
> are several very reputable scholars who have stopped publishing in traditional
> journals.
And what follows from that? That this can be safely extrapolated to all
the reputable and not-so-reputable authors of the annual 2,000,000? Of
course there are always exceptions. If Newton were alive today, he would
have no peers, and we could all safely read every raw manuscript he
wrote. But nothing whatsoever follows from that for the 2,000,000. Such
cases (usually elite ones) simply do not scale!
> Furthermore, some estimates have been made of arXiv submissions,
> and a noticeable fraction of them do not get submitted to journals. (I can
> testify to some really outstanding papers in mathematics that are available
> only through arXiv, because their authors simply never bothered to submit
> them to journals.)
This is partly the Newton effect again (for the elite at the peak of the
distribution). As to the noticeable fraction never submitted to
journals, we need to know a bit more:
(1) How big a fraction?
(2) Apart from the Newton-subset (how big a subfraction is that?),
how does this fraction compare in quality to the the rest of the papers
(i.e., those that did go on to appear in journals)?
(3) Were some (many) of these rejected by journals? revised and
resubmitted under another name? incorporated instead in later work that
was eventually accepted by a journal?
Such questions are very interesting, and we too have done and are doing
such analyses --
http://opcit.eprints.org/tdb198/opcit/ --
http://opcit.eprints.org/ijh198/ --
but these data are certainly no substitute for (or even predictive of
the outcome of) testing what would happen in the complete absence of
classical peer review (as sketched above).
> Part of our difference is probably rooted in our varying professional
> experiences. As just one example (there are others in "Tragic loss ...")
> some of the most interesting developments in mathematics over the last
> half century were in algebraic geometry (leading to several Fields Medals,
> the mathematics equivalent of a Nobel Prize). Much of that work was
> based on several thousand pages published in the un-refereed Springer
> Lecture Notes in Mathematics (the famous SGA volumes). Why was the
> field willing to rely on those? Well, it is a long story, but there
> were enough checks around (such as many people, including graduate
> student seminars going over those volumes line by line) to convince
> experts to rely on those papers. It was examples like these, definitely
> involving review by peers, but not classical peer review, that helped
> convince me that scholars could thrive under a variety of systems.
> I spend an inordinate amount of time on this subject in "Tragic loss ..."
I read it, and found it very interesting, but it does not test the
hypothesis. It merely confirms that classical peer review can be
supplemented in various ways (especially with elite researchers,
highly specialized topics with relatively few practitioners and known
to one another, and relatively fast-moving developments). It certainly
does not follow that these special-case supplements can serve as
substitutes for classical peer review, nor that they scale to research
as a whole.
Here is a logical reductio (since so much of this is a question of
scale): If research just consisted of algebraic topology, and there were
only 6 algebraic topologists doing all the breakthrough work, we would
not need journals and peer review: We could revert to the exchange of
learned letters of Newton's day. Unfortunately (perhaps), this does not
scale to the full gaussian distribution of fields and quality represented
by the 2,000,000 annual papers published in our day.
Classical peer review evolved specifically to cope with the increase in
scale of research. Advocating posting it all to the skies and assuming
that vetting will somehow take care of itself at this scale strikes me as
unrealistic in the extreme.
In some ways, it is a pity that self-archiving and open-access began
with unrefereed preprints. It gave two wrong impressions (diametrically
opposite ones, in the event): Some people concluded, wrongly, that
self-archiving and open-access are only suitable for unrefereed
preprints, whereas refereed postprints should be toll-access. (I think
we all agree that that is not only wrong, but nonsense.) Others
concluded (equally wrongly), that unrefereed self-archived, open-access
preprints are all we need: We can dispense with the peer review and the
postprints.
How much better it would have been (but alas it is too late to redo it)
if the first ones who had "twigged" on the fact that open-access is
optimal for all research had self-archived instead their refereed
postprints, rather than their unrefereed preprints. The reason this
did not happen is that (1) most researchers then (and now) wrongly
believed that copyright law prevents them from self-archiving their
refereed research and (2) the physicists, though not silly enough to be
deterred by such copyright worries, were focussed mainly on the "growing
edge" of their work, which precedes the refereed postprint by 8-12
months. So they took to self-archiving their pre-refereeing preprints
first (though from the outset, many swapped or added the refereed
postprint 8-12 months later, once it was available).
Today it is still copyright worries that are holding back self-archiving
(except among the much more sensible physicists and mathematicians). But
a further worry has been added to retard self-archiving: that it might
destroy peer review, and hence the quality and navigability of the
research literature. And this worry has been (needlessly) encouraged by
the (incorrect) interpretations the self-archiving physicists and
mathematicians have made of what they have actually been doing, and what
follows from it. They THINK they have shown that peer review is not
necessary; in reality what they have shown is that open access is
optimal.
If self-archiving had (mirabile dictu) begun instead with refereed
postprints, we might have spared ourselves these misconstruals, and we
might have been further along the road to open access by now....
>sh> Peer review is not a gold standard, but I'm sure you will agree
>sh> that any alternative would have to ensure at least the same
>sh> standard, if not better: Do you think there is this evidence
>sh> for the promise you are holding out above?
>
> Yes, I do. Examples such as that of algebraic geometry (mentioned in my
> previous response above) showed me early on that there is nothing sacred
> about classical peer review. The law review system is yet another example.
I'm afraid we will have to agree to disagree about that. I have tried to
explain why those cases do not test the hypothesis, and what tests are
needed.
>sh> I honestly can't see how you imagine this scaling to the annual
>sh> 2, 000,000 papers that currently appear in the classically
>sh> peer reviewed journals! Absent the peer reviewed journal, how
>sh> can I know that a paper has been "vetted by experts of a top
>sh> caliber"? What tells me that (as the journal-name currently
>sh> does) for those 2,000,000 annual papers? And what now gets
>sh> the right-calibre experts vetting the right papers (as editors
>sh> formerly did, when they invited them to referee?). Do experts
>sh> voluntarily spend their precious time trawling the continuum of
>sh> raw papers on the net on their own?
>
> I do not have the time to respond to all these points, but yes, many experts
> do "voluntarily spend their precious time trawling the continuum of raw papers
> on the net on their own." I do know many people whose day starts with a scan
> of the latest arXiv submissions. Moreover, some put a lot of effort into
> making what they find more easily digestible for others. (John Baez and his
> wonderful "This week's finds in mathematical physics" comes to mind.)
I can only repeat: This is all parasitic on a classical peer review
system that is still intact behind all of this. It does not and cannot
tell us what would happen without it.
> I am sorry, Stevan, but you are ignoring some of the most interesting
> evolutionary developments in scholarly publishing in your blind faith
> that classical peer review is the only thing that stands between us
> and chaos.
Perhaps. But it is a historical and evolutionary fact that classical
peer review is still 100% intact and in place behind all these
developments, which therefore makes them supplements to peer review,
not substitutes, until such a time as evolution or experimentation
actually tests them as substitutes.
> Open archives are not a religion to me, just a step towards
> a better scholarly communication system, which will also require changes
> in peer review.
I don't think open-access is a religion for me either (though it has
become a bit of an obsession). For me too, it is merely a means to
an end. The end, though, is the enhanced impact and interactivity, hence
productivity, of research. I don't think it's the flaws of classical
peer review that are holding that back (especially now that
pre-refereeing preprints can be made openly accessible too), but the
access-barriers of the toll-access system. Open access to it all would
solve that problem -- and it would also open the door to any
evolutionary developments along the line you are hypothesizing, if they
turn out to be adaptive -- but only after needless worries about
copyright and peer review are overcome. Our point of disagreement is only
about the advisability of needlessly exacerbating worries about peer
review while self-archiving and open access still have not prevailed.
Stevan Harnad
Received on Wed Nov 06 2002 - 13:50:28 GMT