Re: Self-Selected Vetting vs. Peer Review: Supplement or Substitute?

From: Andrew Odlyzko <odlyzko_at_dtc.umn.edu>
Date: Wed, 6 Nov 2002 22:25:30 -0600

Please feel free to post our exchange in the American Scientist Forum.

> (R1) Your primary motivation is (R) to reform research communication,
> mainly through the posting of all papers online, unrefereed, and then
> relying on self-selected "vetters," in place of of classical peer review,
> to evaluate, improve and thereby sign-post paper quality.

I would not phrase it this way. I would say that my primary motivation
is indeed to reform research communication, mainly through the posting
of all papers online, unrefereed, and then relying on whatever mixture
of classical peer review, or contributions of self-selected "vetters,"
the community decides to rely on. I do not hold dogmatic views of how
peer review will be handled, and predict only rough trends.

> (A1) My primary motivation is (A) to make the classically peer-reviewed
> research literature we have now openly accessible online, such as
> it is, to maximize its accessibility, usage and impact. Qualified
> journal editors continue to select the referees, referees and author
> remain answerable to the editor, and the journal-name and its quality
> track-record sign-post for potential users that an article has met
> that journal's quality-standards.

> (R2) You see a causal connection between A and R: A will lead to R.

Yes, provided that (A) involves open access to preprints. (The way you
have phrased (A), it could encompass a system in which the Ingelfinger
rule would be universal, it's just that after publication by a journal,
articles would be freely available. That form of (A) would not yield
to (R).)

> (A2) I see a causal connection between worries about R and not-A:
> Worries that R would compromise or destroy the quality and usability
> of the peer-reviewed literature hold researchers back from doing A.

Yes, agreed.

> (R3) You think peer review is flawed and should be replaced by something
> better.

by a better form of peer review, with occasional non-peer review having
some impact as well.

> (R4) You think that the research advances that occur before peer review
> through the online posting of pre-refereeing preprints today are evidence
> that peer review is unnecessary and can be replaced by spontaneous vetting
> (R) without loss (and perhaps even with a gain) in quality.

Not that "peer review is unnecessary," but that "classical peer review is
unnecessary." I would also quarrel with the phrasing of the last part
of this point, but will let it go for lack of time.

>ao> We do have substantially different visions of the future of peer
>ao> review, and they have not changed much since we first started
>ao> corresponding back in 1993.
>
> I agree. Nor has the evidence changed since 1993.

But there is much more of it now.

> There is not only this total empirical gap between the data you use and
> the conclusions you draw, but there are also logical gaps: You have not
> replied when I have asked how, in a system where classical peer review
> and journal-names with track-records are no longer there as the back-up
> and bottom line -- as they are universally and without exception now --
> how the annual 2,000,000 papers (which are today refereed and sign-posted
> by refereed journals) will find their proper vetting, and how this will
> be sign-posted for potential users? This question does not even come up
> in the case of pre-refereeing preprints, because those are a "parallel
> economy," backed up by the classical peer-review and then sign-posted by
> the names and track-records of the journals to which just about every
> one of those preprints has been submitted, and in which they will all
> appear eventually (though perhaps only after several rounds of revision
> and refereeing, and perhaps not always at the level of the journal to
> which they were submitted first.)

I did not reply because I did not have time to reply to all of your points.
Since you make this such a central point, let me respond now (although very
briefly and so inadequately). How will all those papers "find their proper
vetting"? Well, how do they find proper refereeing now, under your vaunted
classical peer review? We know that serious frauds like the Jan Hendrik
Schoen slip through. We also how plenty of evidence that lots of simply
not very solid science that is not fraudulent gets through. (I don't have
time to dig up references, but there was a paper quite a while ago that
looked at the statistical methodology used in a large sample of medical
papers. It found a horrendously high rate of misapplications of statistics.
There are lots more examples.) The point is that classical peer review does
not provide much of a signal, especially for journals in the lower quality
tiers. So how does science progress? Well, there are all sorts of checks
that are applied post-publication. (And none of them are infallible. Even
a few Nobel prizes are now regarded as having been given in error.) Basically
classical peer review is just one noisy and uncertain signal that the
scholarly community relies on.

> If self-archiving had (mirabile dictu) begun instead with refereed
> postprints, we might have spared ourselves these misconstruals, and we
> might have been further along the road to open access by now....

The incentives were not there to do this. The authors, who after all control
the information flow, could see the benefits to themselves of quick circulation
of preprints. Open access to published journal articles was of much less
value to them, since they typically had access to those journals in their
libraries.

Andrew Odlyzko
Received on Thu Nov 07 2002 - 04:25:30 GMT