Re: Peer Review Reform Hypothesis-Testing

From: Albert Henderson <chessNIC_at_COMPUSERVE.COM>
Date: Mon, 25 Feb 2002 17:36:00 -0500

On Sat, 23 Feb 2002, Sergio Della Sala and Jordan Grafman wrote:

> We are convinced that peer-review is central to scientific credibility.
> However, as it stands the process is far from watertight. Is there any
> way we can improve it by suggesting any modification, either radical or
> minimal? Time is ripe for such a discussion to be launched (see the
> JAMA and BMJ four congresses on peer review in biomedical publication:
> www.jama-peer.org).

        Many papers presented at the JAMA/BMJ congresses and
        other sources point to a problem that I call
        "insularity." That is ignorance of, ignoring or
        avoiding inconvenient information. This includes
        national and language biases as well as the sort of
        short-sightedness that led to the death of a subject
        at Johns Hopkins last year and commercial biases that
        typically omit studies that contradict the desired
        conclusion.

        To combat insularity, several medical journals adopted
        the Consolidated Standards of Reporting Trials [CONSORT].
        One of the recommendations was that authors "state
        general interpretation of the data in light of the
        totality of the available evidence." A study at the
        Prague conference showed little evidence that authors
        complied or that authors were able to compel it. The
        little time reportedly spent by referees, according
        to other studies, suggests they would not catch
        many omissions and blind spots. The Achilles' Heel
        of peer review is that referees are no better informed
        than authors.

        Of course, it is the sponsors of research who call
        the tune. The totality of the literature is
        overwhelming. That includes not only primary reports
        but review articles. The sponsors appear to tolerate
        a shallow review in proposals and preparation, and
        little more in conclusions.

        More intensive screening, evaluating, digesting, and
        review of all lines of research is essential. Many
        reviews reflect an erroneous consensus, such as the
        notion in the 1940s that research on steriods was at a
        dead end.

        I have written more on this in SOCIETY 38,2 47-54
        (J/F 2001), if anyone is interested. I would also
        be happy to provide references to studies of peer
        review that actually shed light on the problem and
        its solution.

        Best wishes,

Albert Henderson
Former Editor, PUBLISHING RESEARCH QUARTERLY 1994-2000
<70244.1532_at_compuserve.com>
Received on Mon Feb 25 2002 - 23:04:03 GMT