On Sat, 17 Feb 2001, Greg Kuperberg wrote:
> sh> Although it is probably a better idea not to draw still further
> sh> attention to this thesis (that self-archiving is a SUBSTITUTE
> sh> for refereed publication, rather than just a SUPPLEMENT),
> sh> because of the confusion and opposition it understandably elicits,...
>
> gk> The claim that e-print archives might have no direct effect
> gk> on peer review is a hard sell for BOTH believers and skeptics
It does and will have an effect, but mostly on the cost and efficiency
of implementing peer review (online is cheaper and faster, especially
if the submission can just be downloaded by the journal from an Eprint
Archive URL, as we are doing with the paper journal I edit:
http://www.bbsonline.org/).
I believe online processing will also shore up some of the weaknesses
of peer review, by putting referee selection on a much broader, hence
potentially more equitable basis, allowing editors to search for and
select qualified referees from a much larger potential pool. And
shortening turnaround times can only help too, in many ways. The
pre-refereeing feedback on the preprints will also be on a larger scale
and faster than it was in paper days, so that will improve papers too,
and help speed the process of convergence on an acceptable draft.
But that is all part of the normal evolution of peer review
(implementation) anyway. It has always taken advantage of new media
(xerox, fax) to try to do things faster and more efficiently. That is
not at all the kind of fundamental and speculative change that is being
advocated without supporting evidence by the would-be "reformers"
(including the outright eradicators) of peer review..
> With the benefit of experience, the objection to your claim can already be
> read from your own terminology. How can you call peer review "a priori"
> if it comes after contribution to the arXiv, which is so much better at
> distributing papers than journals? What is it prior to?
The answer is very simple:
(1) There is, and always has been, "a priori" feedback from colleagues,
prior to peer review. This is informal, nonbinding, and self-selected
feedback. It was solicited by authors on-paper, it continues to be so
on-line, on a wider scale, as noted above. THIS IS NOT PEER REVIEW!
(2) In peer review, it is not the author who self-selects his referees,
nor the referees who self-appoint themselves. The system is a BINDING
and ANSWERABLE one, implemented by a (supposedly) competent, qualified,
even-handed, expert Editor (or Associate Editor, or Board). It is the
Editor who selects the referees and which of their recommendations will
be binding; and the author is answerable to that editor and the
selected referees.
Of course the Editor is and always has been the weak link in peer
review. If an incompetent or biassed Editor is in charge, things can go
badly wrong. But fortunately, at least in science, the system is
self-corrective, and incompetence and bias are eventually ferreted out
(by authors, readers, referees) -- or at least so one hopes (and no one
has demonstrated a better way so far). Journals' current (hierarchical)
level of quality (such as it is) is a result of the degree to which
this system is indeed working. If there is a better way, it first has
to be tested and shown to work.
Peer review is neither self-policing nor vigilantism. It is quality
control of expert work, answerable to qualified expert editors, who
select qualified expert referees. The editors themselves are in turn
answerable to the peer community, for the quality of the contents of
their journals.
> When I write a new paper, I first arXiv it and then I submit it for
> peer review. Empirically peer review for me is already "post hoc",
> whether it extends an "invisible hand" into my research or not.
We are right now doing a survey so we can weigh this kind of anecdotal
observation on a population rather than an individual basis, and
can compare what Simon says, and why, with what Simon actually does.
Please ask your colleagues participate in the survey:
http://www.eprints.org/survey/
But the short answer is: There has always been nonbinding a-priori
informal feedback AS WELL AS binding formal feedback. How much a
particular paper changes as a result of the one or the other is no
doubt an individual matter; the survey will try to ascertain what
authors FEEL is actually the case.
How much the overall quality of the outcome is a consequence of the
ubiquitous constraint of answerability (the "invisible hand" effect)
will only be known when someone tests it empirically. NO ONE HAS DONE
SO. (And, as I repeat endlessly, the archive status quo is NOT a test
of the alternative hypothesis; everything is still submitted for peer
review and 100% answerable to it, just as it always was. No new causal
inferences can be made from that. [I should think that a little
reflection would make this clear to mathematicians, even though theirs
is not an empirical but an a-priori science; in any case, some
self-archiving physicists have been making the same unwarranted
inference, and theirs is!])
> Even if you don't like me giving myself as an example, it is certainly
> not true that "everyone keeps submitting everything for refereeing".
> I gave an example before: The arXiv article q-alg/9709040, by Maxim
> Kontsevich, isn't published. This is hardly a forgotten, rejected draft
> either. Kontsevich is famous, and the result in that article is one of
> his crowning achievements. Indeed, q-alg/9709040 has earned a far more
> prestigious form of peer review than journal publication, since it was
> read and recognized by the Fields Medal committee. Prestigious though
> it may be, it is also "post hoc" from any viewpoint.
Alas, it is hard to draw empirical conclusions from an N of 1...
> I have never interviewed Kontsevich to get his opinions on scholarly
> communication. I think that he isn't interested in that topic and
> that he harbors no particular skepticism about journals. He does still
> publish most of his papers. But I also believe that Kontsevich hasn't
> gotten around to publishing q-alg/9709040 because he has been from the
> yoke of promotion. My theory is that he might publish it somewhere if
> he had some free time, but he's busy with new research, and anyone who
> wants it can get it from the arXiv.
I hope the survey will give us a bigger N to base conclusions on
(though of course it will have the sampling bias that always comes form
self-selection!)
--------------------------------------------------------------------
Stevan Harnad harnad_at_cogsci.soton.ac.uk
Professor of Cognitive Science harnad_at_princeton.edu
Department of Electronics and phone: +44 23-80 592-582
Computer Science fax: +44 23-80 592-865
University of Southampton
http://www.ecs.soton.ac.uk/~harnad/
Highfield, Southampton
http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM
NOTE: A complete archive of the ongoing discussion of providing free
access to the refereed journal literature online is available at the
American Scientist September Forum (98 & 99 & 00 & 01):
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html
You may join the list at the site above.
Discussion can be posted to:
american-scientist-open-access-forum_at_amsci.org
Received on Wed Jan 03 2001 - 19:17:43 GMT