Re: Self-Selected Vetting vs. Peer Review: Supplement or Substitute?

From: Andrew Odlyzko <odlyzko_at_dtc.umn.edu>
Date: Wed, 6 Nov 2002 09:14:17 +0000 (GMT)

Just a couple of brief responses to your comments. Beyond this,
I do not intend to continue. We do have substantially different
visions of the future of peer review, and they have not changed
much since we first started corresponding back in 1993.

> ...if your prediction happens to be wrong, making the prediction
> anyway will have a negative, retardant effect on self-archiving and open
> access. (Of course, if you are right, then concerns about these changes
> will still have a negative, retardant effect on self-archiving.)

I am not sure it will be negative. It might encourage the transformation
by showing the path to a future in which not only information dissemination,
but also peer review, are improved.

>ao> How does this differ from somebody a decade or two ago that
>ao> might have promised that electronic publishing would simply mean
>ao> that journals would now be available online, but there would
>ao> be no disturbing innovations such as scholars being confused by
>ao> uncontrolled preprint distribution?
>
> I can't see the point (and I'm not sure what you mean by scholars being
> confused by uncontrolled preprint distribution!).

Some of the opposition to electronic publishing was based on concerns about
uncontrolled proliferation in available material.

> (Nor does it seem to me that you are making
> these predictions because you are recommending that people think twice
> about the transition, or first take some remedial measures.)

Certainly not. I hope to hasten the transition, by pointing out that
it is likely to improve the peer review system.

> Self-archiving was tried extensively, and demonstrated to work. Now we
> can confidently say it works and recommend it to everyone. Alternatives
> to peer review have not been tried or demonstrated to work. Nothing
> prevents people from trying to implement controlled experiments on
> alternatives to peer review. But until they are done, and the outcome
> known, there is no basis whatsoever for linking them to self-archiving
> and open-access.

Sorry, but alternatives to classical peer review have been tried, and
are constantly being tried (quite successfully, in my opinion). That is
largely what my article "The fast evolution of scholarly communication"
was about.

> The fact, though, is that no one KNOWS that your prediction is true. So
> dilating on it now -- when its truth cannot even be known, and when,
> on the face of it, proclaiming it will merely reinforce people's fears and
> hesitations about self-archiving -- can hardly serve a useful purpose. (If
> I were you, and I could not in good conscience deny my belief in the
> causal connection between self-archiving, open-access, and the changes in
> peer review that you described, I simply would not express my belief at
> all, rather than risk voicing a fallible belief that is almost certainly
> going to have a negative effect on something I regard as very positive,
> but also certain.)

Well, yes, we don't know whether my prediction is true, but there is
evidence for it, for example in "The fast evolution ...." Peer review
is undergoing change right now, in front of our eyes, even though
few are paying attention. To pretend that nothing will change seems
really short-sighted.

> ...the peer review generates a reliable,
> recognizable, quality-level-tag, a recognizable milestone with an
> established track record, along the continuum, on which the would-be user
> can depend. Without that, it is not at all clear where a particular paper
> stands, in quality and usability, along its own continuum...

But even with classical peer review "it is not at all clear where a particular
paper stands ..." We get a very weak quality and usability signal from
classical peer review, and my contention is that we can obtain many other
signals that collectively, if not individually, can be even more useful.

> Reinvented or reshaped where, and by whom? As we speak, whether a
> self-archiver or not, not a single author of the annual 2,000,000 papers
> that appear in any of the hierarchy of 20,000 peer-reviewed journals
> published across all disciplines and around the world has stopped
> submitting his papers to those journals. Your predictions are merely
> speculations. They have not been implemented and tested, and what the
> outcome would be if they were tested is not known.

I would have to go back and dig up some old messages, but I believe there
are several very reputable scholars who have stopped publishing in traditional
journals. Furthermore, some estimates have been made of arXiv submissions,
and a noticeable fraction of them do not get submitted to journals. (I can
testify to some really outstanding papers in mathematics that are available
only through arXiv, because their authors simply never bothered to submit
them to journals.)

Part of our difference is probably rooted in our varying professional
experiences. As just one example (there are others in "Tragic loss ...")
some of the most interesting developments in mathematics over the last
half century were in algebraic geometry (leading to several Fields Medals,
the mathematics equivalent of a Nobel Prize). Much of that work was
based on several thousand pages published in the un-refereed Springer
Lecture Notes in Mathematics (the famous SGA volumes). Why was the
field willing to rely on those? Well, it is a long story, but there
were enough checks around (such as many people, including graduate
student seminars going over those volumes line by line) to convince
experts to rely on those papers. It was examples like these, definitely
involving review by peers, but not classical peer review, that helped
convince me that scholars could thrive under a variety of systems.
I spend an inordinate amount of time on this subject in "Tragic loss ..."

> Peer review is not a gold standard, but I'm sure you will agree that
> any alternative would have to ensure at least the same standard, if
> not better: Do you think there is this evidence for the promise you are
> holding out above?

Yes, I do. Examples such as that of algebraic geometry (mentioned in my
previous response above) showed me early on that there is nothing sacred
about classical peer review. The law review system is yet another example.

> I honestly can't see how you imagine this scaling to the annual
> 2,000,000 papers that currently appear in the classically peer reviewed
> journals! Absent the peer reviewed journal, how can I know that a paper
> has been "vetted by experts of a top caliber"? What tells me that (as
> the journal-name currently does) for those 2,000,000 annual papers? And
> what now gets the right-calibre experts vetting the right papers (as
> editors formerly did, when they invited them to referee?). Do experts
> voluntarily spend their precious time trawling the continuum of raw
> papers on the net on their own?

I do not have the time to respond to all these points, but yes, many experts
do "voluntarily spend their precious time trawling the continuum of raw papers
on the net on their own." I do know many people whose day starts with a scan
of the latest arXiv submissions. Moreover, some put a lot of effort into
making what they find more easily digestible for others. (John Baez and his
wonderful "This week's finds in mathematical physics" comes to mind.)

I am sorry, Stevan, but you are ignoring some of the most interesting
evolutionary developments in scholarly publishing in your blind faith
that classical peer review is the only thing that stands between us
and chaos.

> It is up to you, but I do not understand why your conscience tells you
> you need to share your speculations (especially when they risk alienating
> the majority who are still leery about self-archiving!).

Well, I was invited to contribute my thoughts on peer review to a book
devoted to the subject, so I did the best I could within the allotted
limits. Open archives are not a religion to me, just a step towards
a better scholarly communication system, which will also require changes
in peer review.

Andrew Odlyzko
Received on Wed Nov 06 2002 - 09:14:17 GMT