On Wed, 16 Oct 2002, [Identity removed] wrote:
> Dear Dr. Harnad--I have been doing a web search on peer review in
> connection with an article I am doing on the journal [deleted].
>
> [Deleted] published the paper but later on said it erred and while not
> retracting the paper said it never should have published it. It arrived at
> that decision after essentially having sent the authors back to the lab to
> verify their findings. What I am exploring is whether an improved
> refereeing procedure might have avoided the flap...
Although it sounds serious and important, in reality scientific fraud
is not a real problem for the simple reason that science is cumulative,
hence self-corrective. Consider these two alternatives:
(1) If a finding is not important enough for other researchers to bother
to try to use it and build upon it, then it hardly matters if it is
erroneous, because it has no effect on further work.
(2) If a finding is important enough for other researchers to try to use and
build upon, and it is erroneous, then it will collapse under the weight
of further attempts to build on it (because it is erroneous).
The only thing that might go undetected is a false NEGATIVE finding,
but, first, there is not much motivation to publish negative findings
(let alone false ones); and second, if the potential positive finding
was promising and important, others will want to test whether it really
was a nonstarter (and will find that it is not). Self-correction again.
Nor is peer review in general to be blamed for the occasional failure to
detect error or fraud: Peer review is simply qualified experts
evaluating the work of other qualified experts. Occasionally they will
err; occasionally they will be biassed. But there is no "system" to
protect against such human foibles as long as the judgments depend on
human minds. At best, journals can use more referees -- but refereeing
is done for free, using precious time that referees poach from their
research, so it is unlikely that this overharvested resource can be
harvested even more intensely. Besides, more referees for everything
would simply mean doing in parallel -- for everything -- what would
be detected serially (through the self-corrective process I mentioned)
for the minority of undetected errors and fraud.
People often speak about the need to reform the peer review system, but
they usually don't realize that there is no peer review "system": it is
merely human experts evaluating one another's work. It's hard to imagine
who could be better qualified to do so than qualified experts (referees),
selected by qualified experts (editors), but if someone has a hunch about
a better system, it would be best if it were tested first, to see whether
it really worked at least as well as the classical system, before being
advocated or adopted.
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/1169.html
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/0479.html
On the other hand, there are many ways that the new online medium can
be used to make peer review more efficient and equitable, to select
qualified experts more broadly and evenly, to distribute the load more
widely, and to speed the process:
Finally, about "retractions": A journal is not a law court. There is no
way, after a work has appeared, to make it disappear or unappear.
Journals stay on shelves. The only thing that can be done if there has
been an error is to make that known: I would say that [Deleted]'s saying
that the paper in question had been in error and that it should never
have been published more or less accomplishes this public corrective
function. What do you think a "retraction" would add (or mean)?
Here too, the online medium is preferable, because it allows corrigenda
and updates to be attached to papers directly, so all users see that
there have been corrections or additions. One cannot do with paper
articles on shelves.
Below is a relevant recent posting on this from Peter Suber:
http://www.earlham.edu/~peters/fos/
FOSN for 5/23/02, linking the retraction problem to FOS issues,
though loosely:
* In the May 20 _Tech Central Station_, Howard Feinberg reports
on the survival of bad scientific ideas after their retraction or
invalidation. A 1998 study by John Budd showed that 235 scientific
articles "retracted due to error, misconduct, failure to replicate
results or other reasons" had been cited 2,034 times after their
retraction, and that most of the citing papers did not mention the
retraction. Feinberg uses the Budd study to set up a discussion
of the recent fiasco at _Nature_, in which a paper was withdrawn
after publication by the editors who faced intensive lobbying both
scientific and non-scientific. (PS: Will FOS aggravate the problem
of overlooking retractions, by keeping old studies circulating
forever in the Google cache and Wayback Machine? Or will it
mitigate the problem, by allowing more intelligent searching and
indexing?)
http://makeashorterlink.com/?E1ED238E
Harnad, Stevan (1985) Rational disagreement in peer review. Science,
Technology and Human Values, 10 p.55-62.
http://cogprints.soton.ac.uk/documents/disk0/00/00/21/28/index.html
Harnad, S. (1996) Implementing Peer Review on the Net: Scientific
Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby,
G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA:
MIT Press. Pp 103-108.
http://cogprints.soton.ac.uk/documents/disk0/00/00/16/92/index.html
Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review,
Peer Commentary and Copyright. Learned Publishing 11(4) 283-292.
http://cogprints.soton.ac.uk/documents/disk0/00/00/16/94/index.html
Harnad, S. (1998/2000) The invisible hand of peer review. Nature
[online] (5 Nov. 1998)
http://cogprints.soton.ac.uk/documents/disk0/00/00/16/46/index.html
Best wishes,
Stevan Harnad
NOTE: A complete archive of the ongoing discussion of providing open
access to the peer-reviewed research literature online is available at
the American Scientist September Forum (98 & 99 & 00 & 01 & 02):
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html
or
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/index.html
Discussion can be posted to: american-scientist-open-access-forum_at_amsci.org
See also the Budapest Open Access Initiative:
http://www.soros.org/openaccess
the Free Online Scholarship Movement:
http://www.earlham.edu/~peters/fos/timeline.htm
the OAI site:
http://www.openarchives.org
and the free OAI institutional archiving software site:
http://www.eprints.org/
Received on Fri Oct 18 2002 - 02:52:51 BST