Re: Self-Archiving Refereed Research vs. Self-Publishing Unrefereed Research

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Wed, 5 Mar 2003 22:46:58 +0000

On Wed, 5 Mar 2003, Arthur P. Smith wrote:

> >>There seem to be a few cases of this in bio-medicine in recent years
> >
> >Which?
>
> The placebo-effect for one. A recent review seems to have seriously
> questioned the original "placebo-effect" research.

But placebo effects were always fuzzy, and hardly medical. Medically,
their only role has been as a control for the effects of (putative)
treatments: I give a drug to N patients, a sugar-pill (placebo) to N
patients (double-blind), and I compare the effects. If significantly
more patients get better with the drug than with the placebo, then maybe
the drug really works. If I had just compared the drug patients to
N patients getting no treatment at all, and there was a significant
improvement, then that might have just been the "placebo" effect: The
inevitable proportion of patients who improve either spontaneously
or because of suggestive effects from belief in the drug. (Placebo
research borders on another fuzzy area, hypnosis research, which in turn
grades into a still fuzzier area, namely, parapsychology: telepathy,
telekinesis!)

But none of this is biomedicine! No one ever started selling "placebo"
as such in drug stores, as a medicine (though many medicines sold as
something else are probably just placebos). Placebo, in brief, is
*non-effect* in medicine. The only thing that keeps coming and going
cyclically is the tendency to overinterpret placebo as being something
*more* than just that. Placebo effects are the perfect example of the
kind of effect on which nothing can be built!

> Almost any "twin" study on heritability seems to be met with great
> skepticism now since the abuses of the original researchers in the early
> 20th century were uncovered.

I think this is dead-wrong (as stated), and more in the nature of what
the Swedish journalist might say, one way or the other. Heritability
studies in twins are robust for all kinds of heritable traits (e.g.,
tendency toward heart disease, obesity, alzheimer's). You probably
mean the twin studies on the heritability of IQ, in which some data
were inflated by Cyril Burt. But there the paradoxical result was that
although his data-fudging was discovered -- and for a while discredited
research on the heritability of IQ, and even on the reality of IG itself
-- it turns out that he need not have bothered inflating his data, because
the effect was real! Yes, a significant portion of the variance in IQ is
heritable too, along with heart disease, obesity, and alzheimer's. (Mendel
did similar fudging, yet his effects turned out to be real too.)

So this is not an example of persisting undetected error either: It is
rather an example of the opposite: That the self-correctiveness prevails
after all, even when *real* errors, even deliberate ones, are suspected
and detected -- if the errors happened to be in the direction of the
true effects anyway!

> And there seem to be frequent reports of researchers only publishing
> "positive" results about drugs, under the encouragement of the companies
> funding their research.

That's certainly true. But that's not science; that is the drug
business. In that gray area between hypothesis and hype, nolo
contendere.

> Andrew Odlyzko mentions an interesting phenomenon that has arisen in
> response to the flood of information, much of it of questionable quality
> - the "authority site". You could think of these as another level of
> "peer review", although they tend to be run by just a single individual
> (and consequently somewhat narrowly focused - but that's not that
> different from a highly specialized journal). Should we be moving to
> fund this sort of thing as a replacement for peer review, or in
> addition? Is there some way these should be formalized and included
> within the recognized knowledge infrastructure of our fields? How do you
> ensure the trust-worthiness of those responsible for these efforts (and
> how exactly is that different from trusting peer review at a regular
> journal?) What aspects of the current information infrastructure for
> scholarly communication help, or hinder, the work of the people creating
> these sites?

As I have pointed out repeatedly to Andrew and Greg Kuperberg in
this Forum, these are all merely *supplements* to peer review,
not *substitutes* for it, because the self-archived preprints are
virtually all submitted for peer review, as they always were. At best,
these authorities provide a faster clearing-house for some papers than
peer-review; and they also apply further levels of validation, *after*
peer review.

These valuable new features of an open-access literature will no doubt
grow and become more systematized and even institutionalized with time,
but they will not replace the basic filter of peer review. They will
merely enhance it. No "authority" has the time to sample and filter
everything, or even most things. If they really did have nothing better
to do with their time than to process all those raw, unfiltered
manuscripts, one would have to begin to wonder about their
competence! Never happens, though, as the "authorities" only vet a
selected subset of the research glut.

But that is neither here nor there for this Forum, whose main objective
is freeing access to peer-reviewed research, not freeing research from
peer review.

Stevan Harnad
Received on Wed Mar 05 2003 - 22:46:58 GMT