On Tue, 13 Jul 1999, Bob Parks <bparks_at_wueconc.wustl.edu> wrote:
> 1. xxx.lanl.gov has about 100,000 papers and that archive does not seem
> to have reduced the number of journals in physics, nor the quality of
> the scientific literature. Hence we have at least one strong piece of
> evidence that 'free access archiving' will not lower the quality. I
> don't know of any evidence showing that quality has been lowered in
> physics or elsewhere.
>
> 3. xxx.lanl.gov seems to have conditioned its audience to 'filter'
> relevent articles from the large number of submissions. I would guess
> that works much like the usual filtering process that any academics use
> for 'hard copy working papers'.
The filter is even simpler than that: "R" (refereed journal, and perhaps
journal name "JX") and "Author Name."
The peer review proficiency of the journals in question takes care of
the rest.
> 4. When we have citation-linking for all scientific literature
> http://www.princeton.edu/~harnad/citation.html it will be natural
> and easy to 'value' writing - namely by the number of citations (and
> possibly the 'quality' of citations). Such citation criteria are
> already used in promotions and salary (at least in my small biased
> sample). One can argue whether quality is better determined from
> citations than from knowing that two or three referees and an associate
> editor have passed judgement.
Fallacy: Apart from the limitations of citation metrics, the validity
they do have is COMPLETELY parasitic on the fact that the papers in
question are published in peer reviewed journals. The invisible
hand of peer review is behind them. Hence they would implode if that
hand were withdrawn.
Impact factors (like peer commentary) are a SUPPLEMENT TO, not a
SUBSTITUTE FOR, peer review. (And peer review is not a go/no-go filter;
it is an interactive, iterative, corrective process of submission,
feedback, revision, resubmission, etc. between author and peer
reviewers, adjudicated by a competent, answerable peer Editor: This is
no passive numerical filter or box score on any blind metric.)
Harnad, S. (1998) The invisible hand of peer review. Nature
[online] (5 Nov. 1998)
http://helix.nature.com/webmatters/invisible/invisible.html
http://www.princeton.edu/~harnad/nature2.html
> 5. As an economist, I would have to argue that the resources devoted
> to refereeing are misallocated because they are not compensated
> directly. In the current journal model, there may be too much
> refereeing (or there may be too little).
Peer review, like democracy, is not without its imperfections. And no
doubt there exist ways to improve it. But those ways must first be
found and tested and demonstrated to improve it -- especially the most
radical one, of abandoning it entirely, in favour of self-archiving of
unrefereed preprints alone.
Note that I do NOT advocate the latter: I have always aimed
(subversively) at the self-archiving of the REFEREED paper, not just
the unrefereed one. That will be tantamount to freeing the current
peer-reviewed literature, SUCH AS IT IS (warts and all).
http://www.arl.org/sc/subversive/
We can worry about ways of fixing the warts independently; but the
powerful and proven benefits of self-archiving should NOT be linked in
any way to speculative and untested notions for improving peer review --
least of all abandoning it altogether.
> If 'free access archiving'
> means the end of journal refereeing as we know it, I am not sure
> whether I (at least) could argue that there is a social gain or loss.
There is every reason to believe it would be an enormous loss, throwing
the baby (a reliable research literature) out with the bathwater (access
barriers to that very literature).
To put it another way, it's READER-ACCESS barriers that have to go;
AUTHOR-ACCESS barriers (into the certified refereed corpus) must stay.
> Referees might spend their time writing/reading rather than refereeing,
> which could result in better scientific literature than what exists
> with their time spent refereeing. I am not arguing that refereeing has
> no value, only that we do not know what that value is, and that
> whatever that value is, it is not compensated (directly at least).
We know the value of the refereed literature, such as it is; every
editor knows what raw submitted manuscripts look like (90% of which will
be rejected, if it is a high quality journal; and most of the 10% that are
finally accepted will look nothing like their initial drafts, for they
will have gone through the iterative corrective feedback cycle of peer
review mentioned above).
It is the difference between these two literatures that is at issue (and
the difference is even greater than that, because even those raw
submissions are prepared with the PRESUMPTION of answerability to peer
review -- yet another manifestation of its "invisible hand").
No, human nature being what it is, without answerability it quickly
regresses toward the anarchic levels of the chat-groups NetNews, that
Global Graffiti Board for Trivial Pursuit. (And neither the Pandemonium
of post hoc "peer" commentary nor the still poster-hoc feedback from
"citations" can provide that answerability. Pity the reader who has to
navigate a chaotic corpus like that.)
> IMHO, the only reason to sort it out is to determine, given the goals
> of the esoteric author (a term I like), whether 'free access archiving'
> will lower or raise the quality of scientific literature.
IMHO, it would not be good empirical practise to test whether the FDA is
really protecting our healths by scrapping it and seeing what happens!
The goal of self-archiving is to free the refereed literature from
access barriers, not to free it from refereeing!
The way to test variants on or alternatives to peer review is locally,
not globally: Do you know of any local experiments? I do, and as far as
I know, it is not yet faring too well!
http://www.bmj.com/cgi/shtml/misc/peer/index.shtml
> Again, the point should be whether the quality of the scientific
> literature is harmed by 'free access archiving'....
> In the NO-JOURNALS world of 'free access archiving'
> we write to attract others attention, and citation. Rather than
> writing for three people (two referees and an editor) we now have to
> write for a larger audience and have to write to attract a readership
> (rather than attract an editor/referee). I don't see that deters us
> writing. The goals of fortune and fame remain, its just the journals
> no longer have a Faustian GRIP on us.
And it all becomes a vanity press, with no sign-posts for the poor
reader and user as to what, in all this unregulated soup, is fit for
consumption!
> The current business 'model' for scientific literature is, well,
> absurd. Editors are mostly not directly compensated, and those who are
> are not compensated at the market value of their time. Referees are
> not compensated ($35 or $50 is not compensation). Authors are not
> compensated at all directly.
Correct. But it is also what vouchsafes us our current refereed
literature, such as it is. Let us free THAT before toying with any
notional improvements -- INCLUDING referee payment, which is
potentially corruptive: They referee for free now, and that's just part
of the system, such as it is.
The system, with its authors giving their papers away free, and its
referees giving away their services for free, is best described as
ANOMALOUS, not ABSURD. What is absurd is to continue treating it
according to the access-blocking trade-model, instead of freeing it
(from S/L/P) for the reader as well.
Here is an argument that could be invoked against me, but I don't think
it's valid:
"Fine, I take you at your word. 'Don't tamper with the system, just
free it.' Now I will show that that admonition is
self-contradictory: I agree that altering or abandoning refereeing
would be tampering, and would put quality at risk. I agree that
paying referees would be tampering, and would put quality at risk.
But then, isn't author self-archiving tampering too, and putting
quality at risk? Might it not bring down the entire system,
destroying the revenue base on which the quality control is built?
Is it, therefore, not one of those untested "reforms" of peer
review against which you always inveigh? Q.E.D."
My reply is simple: Authors have always given away reprints of their
papers for free. Self-archiving simply increases the scale of this. And
the waters HAVE been tested, for close to a decade now, by Los Alamos,
and no sign of diminished quality has emerged (as Bob notes above).
http://xxx.lanl.gov/cgi-bin/show_monthly_submissions
So the free waters are safe for peer review. (And if and when another
revenue source must be found for continuing to fund it, the up-front
redirection of 1/3 of S/L/P savings to institutional publication costs
is ready as a natural source for it.)
Hal Varian's worries about possible knock-on effects elsewhere in the
quality control system are not entirely without basis, but there doesn't
seem any compelling reason for alarm either. Let self-archiving proceed
apace.
> So the university pays us to
> author/edit/referee and then buys our product back from a 'publisher'.
> Resources must be misallocated in that model. If our current world was
> a 'free access archiving' with citation valuations (rather than journal
> valuations), proposing such a business model would, well, be absurd.
> We need to unshackle ourselves from the current journal Faustian Grip,
> from that mental model of the world, and proceed ahead. Nor should we
> consider that scientific literature fits into other 'information'
> products.
This seems a bit garbled to me, because it conflates freeing the
refereed literature form S/L/P with freeing the literature from
refereeing.
> Much of the discussion between Hal and Stevan side steps into business
> models (ignoring any further words on motivations of authors). So what
> is the business that requires a model? Production of (quality)
> scientific literature.
Actually, the "business" is making an impact on research with one's
research. The literature is just a means, not an end. Hence my analogy
with ads.
> Must that be tied to the elsevier et al (I use
> elsevier in lower case as a generic for profit and non-profit
> presses)? elsevier does not pay the authors, nor the referees nor the
> editors which is 95% to 99.9% of the real cost of producing the
> literature. In the 'free access archiving' world, we do not need to
> wory about whether elsevier survives. We do need to worry about the
> quality of the scientific literature, and elsevier itself does not
> provide that quality control. Editors and referees do. Citations do.
Peer review does; citations do not! Elsevier (and others) implement
peer review. That will always cost something -- but nothing like what
the whole papyrocentric, S/L/P-based system costs now.
> Whether universities are willing to compensate us for editing and
> refereeing without the elsevier label is an open question (especially
> if the citation linking proposal becomes fact). In fact it is a
> question which should be asked - how much refereeing should be done?
> If we have citations, do we need refereeing and editing? It is not
> that refereeing and editing do not increase the value of an article, it
> is whether the correct amount of resources are devoted to that
> activity, and whether citations (or similar) would be a more cost
> effective way to discern the quality (for promotion, tenure, etc.).
Vide supra. It seems to me naive in the extreme to imagine that delayed
post-hoc citations can substitute for the substantive, interactive,
quality-control process of peer review.
> Imagine a world with 'free access archiving' without journals. How
> does one get promoted? Citations and review letters. Citation
> analysis would be free, and universities would have to compensate for
> outside review letters. Would that really change the quality of
> scientific literature - for the worse? Not in my mind.
Have a peek at Usenet/Netnews for a glimpse of where things would head,
human nature being what it is...
--------------------------------------------------------------------
Stevan Harnad harnad_at_cogsci.soton.ac.uk
Professor of Cognitive Science harnad_at_princeton.edu
Department of Electronics and phone: +44 2380 592-582
Computer Science fax: +44 2380 592-865
University of Southampton
http://www.ecs.soton.ac.uk/~harnad/
Highfield, Southampton
http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM
ftp://ftp.princeton.edu/pub/harnad/
Received on Wed Feb 10 1999 - 19:17:43 GMT