> Bruce Edmonds <b.edmonds_at_MMU.AC.UK>
>
> A key difference between Stevan's and my proposals is that in his one
> either gets a generic quality constraint on the flow of information
> (via journals) or none at all (using public pre-print archives). In my
> proposal papers are `tagged' with relatively rich judgemental
> information so that the reader can determine the selection of
> information to suit their own needs - mixing judgemental and
> content-based information. Readers could emulate the journal system by
> choosing selection criteria so as to cover a single topic area allowing
> only papers of high general quality, but they would have the choice!
As we have been back and forth on this before, let me just repeat that
your proposal does not, as it suggests, capture the classical quality
control and tagging system as a special case. It loses it completely,
and offers no tested control/tag system in its place, but rather an
untested one with a number of prima facie reasons, described
repeatedly, to expect that it will fail (both as a way of eliciting
reliable peer review and as a way of filtering and signposting the
literature for the user).
> This early and inflexible constraint on the flow of papers from author
> to reader that occurs in the journal system was necessary because
> publishing used to be costly - this is no longer the case. Reviewers
> already produce this information - why not release it to readers so
> that they can determine their own selection criteria?
Same story: They CAN release it to readers (by publicly archiving their
papers), but with no need either to sacrifice classical peer review in
exchange or to swap it for an untested alternative with many prima
facie problems. As repeatedly stated, virtually all the advantages you
cite are available without following your proposal, simply by public
self-archiving of papers (and, if desired, of referee reports too).
> Yes, I accept that a system which involves a dialogue between author
> and journal is different [from] a one-shot review. However I disagree on
> the assessment of the overall impact of this change. The one-shot
> system would encourage authors to get feedback from colleagues *before*
> submission, thus saving editorial staff a *lot* of time.
Most authors already get advance feedback, by circulating their papers
to experts for feedback before submitting them for publication.
Self-archiving will increase the scope of this presubmission feedback.
Again, you have it without any need for the system you propose.
Submission/refereeing/revision/re-refereeing/revision... is not aptly
glossed as "a dialogue between author and journal"...
> Review boards, could unilaterally decide to review an on-line paper,
> they would not have to wait for submitted papers. Thus there is no
> danger of authors being afraid of the process - they would have no
> choice but to get used to it!
This is a non sequitur, and based on an unrealistic idea about the
journeyman time available to editorial boards and referees,
particularly if they are ever faced with a Global Netful of unfiltered
drafts. Some authors might be happy with exhibitionism and taking
lumps; but the ones really at risk are the referees -- and the readers.
> The balance of work for reviewers is far from obvious.
Not obvious? Currently I review only what an expert editor asks me,
based on my expertise, to review, and with the understanding that the
author is answerable to the review process. Who would review what if
there were just a Netful of raw manuscripts waiting for all comers? And
what is the point of reviewing if there is no answerability beyond the
(temporary) public co-archiving of the reviews themselves, along with
the rest of the unfiltered drafts?
> Considerable
> time would be saved in mark-up and in the management of the review
> process.
Markup has nothing to do with the peer review process. Peer review
assesses and provides (binding) feedback about the quality of the
CONTENT of papers; markup is concerned with tagging FORM.
> On the other hand papers will be reviewed more than once by
> different boards - but this is an appropriate and vital academic
> activity, on a par with writing and reading.
Do not mix up published post-publication reviews (book reviews, formal
commentaries) with peer review. Peer review is a feedback process meant
to help shape the paper's contents to meet the quality criteria of a
particular journal within the quality hierarchy of journals. Multiple
review AFTER publication is nothing like multiple submission BEFORE
publication, which is a huge drain on the finite resources of peer
review.
> I think more effort is
> required judging academic output so that it becomes accessible to more
> than one audience. At the moment this lack of cross-judging impedes the
> flow of information from field to field.
Archive publicly. Then anyone can "cross-judge" it. But don't expect the
same attention for unfiltered self-archived drafts as for quality-
controlled ones, duly tagged by the journal that has done the peer
review. You are conflating two features here.
> Yes there would be many changes to get used to. One would have to put
> up with getting poor reviews on your on-line journal, but if you did not
> want it to be judged, why make it available.
This is purely a single-author-eye's view of the literature, the view
from which most of the unrealistic proposals for new "systems" tend to
be made. The literature is not there only, or even primarily, so as to
vouchsafe an author the maximum amount of attention and exposure; that's
what Hyde Park is for. The literature is there to provide information
for us all, information that we can trust and navigate. And for that it
needs to be quality controlled by experts in advance. Otherwise we all
have to find our way through the unfiltered morass for ourselves.
So it is not sufficient to fantasize about authors being kept up to
standards by fears of receiving negative public reports: In the system
you recommend it is not clear who would bother to read what any more;
so it is equally unclear whether negative public feedback would even be
systematically noticed by anyone in the vast unfiltered pot-pourri of
papers and comments (another kind of paper) that would replace the
refereed literature.
As an exercise, please step through your proposal as it would apply to
papers on cancer treatment, and give some indication of how a
practitioner should apply this literature to his ongoing practise.
Don't focus on the quack paper that is prominently denounced as such;
focus on the vast majority of borderline papers, where it is not clear
whether they are or are not reliable.
And don't reject the biomedical literature as a special case: Some of
us consider it just as important to sort the reliable from the
unreliable in any area of scholarly and scientific research to matter.
> There would be
> compensation in that, with a plethora of review boards, one would get
> used to have a variety of reviews depending on its acceptability and
> importance for different audiences.
Your proposal is like social credit: You are hypothetically disbursing
resources (referee time and effort) that there is no evidence you would
have left if your system were actually adopted (and strong prima facie
reasons to expect you would not).
> Life is `tough' for academics, if they go public with ideas they are
> responsible for the quality, regardless of whether it is formally
> submitted to a journal.
Correct, and, as above, this remains just as true with self-archiving
PLUS classical peer review as with self-archiving MINUS classical peer
review (and some other untested option in its place).
> One has to accept one's work will be judged and
> not release it (i.e. publish it) until one is ready to take that risk.
True again, with or without your proposal.
> It is a *problem* with the current system that the level of judgement is
> low, unless it is formally submitted. But my way you can have this
> judgement _without_ restricting the availability and flow of information
> until it almost reaches the reader.
This sounds incoherent (what is "almost"?). You can self-archive your
paper, and take your lumps publicly. This is true with or without
classical peer review. You can even set up an archive and baptize it an
"open journal," in which authors get "referee reports" and stars
alongside their self-archived papers (though it is not clear who would
do the refereeing or why). That's part of taking the lumps too. But for
those who would prefer to restrict their reading to papers that (at
least) meet existing quality standards, those standards still have to be
applied, by qualified experts, selected by and accountable to expert
umpires (editors), who ensure that only what has met the standards,
possibly after several rounds of revision and re-refereeing, gets
tagged as such. Nothing in your proposal fills this need, or shows that
it need no longer be filled. "Journal" still seems a reasonable name to
reserve for what fills it.
--------------------------------------------------------------------
Stevan Harnad harnad_at_cogsci.soton.ac.uk
Professor of Cognitive Science harnad_at_princeton.edu
Department of Electronics and phone: +44 1703 592-582
Computer Science fax: +44 1703 592-865
University of Southampton
http://www.ecs.soton.ac.uk/~harnad/
Highfield, Southampton
http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM
ftp://ftp.princeton.edu/pub/harnad/
Received on Wed Feb 10 1999 - 19:17:43 GMT