CALL: CORTEX FORUM ON PEER-REVIEW
Cortex regularly publishes a section called Discussion Forum on
science-related issues (see a recent one on The Use and Misuse of Impact
Factors in Cortex 37, Issue 4, 2001; also available on the web, for both
subscribers and non subscribers alike, at:
http://www.cortex-online.org ).
The next Forum will be on The Role of Peer-Review in Neuropsychology. A
preliminary draft of the accompanying editorial is attached for your
perusal.
If you wish to contribute to the discussion on any issue relevant to
peer-reviewing, please let us know (cortex_at_abdn.ac.uk) by March 1st.
Each contribution should be no longer than 1000 words plus 1 figure or table
and no more than ten references (see previous Forum for examples). Deadline
for final submission is: March 29th, 2002.
Further information can be obtained from the editors:
Sergio Della Sala: sergio_at_abdn.ac.uk
Jordan Grafman: grafmanj_at_ninds.nih.gov
Contributions should be sent to:
Cortex
Department of Psychology
William Guild Building
University of Aberdeen
Aberdeen
AB24 2UB
Refereeing mortus est, vivat refereeing
Sergio Della Sala and Jordan Grafman
We are all well aware of some of the shortcomings in the peer-review
process of scientific work. The system has been revealed to be riddled
with prejudice, subject to nepotism (Forsdyke, 1993, Calza and Gerbisa,
1995; Perez-Enciso, 1995; Wenner s and Wold, 1997), sexism (Wallston
and OLeary, 1982; Wenner s and Wold, 1997), and influenced by the
national language of the authors (Bakewell, 1992; Nylenna et al.,
1994), not to mention the problems of broken confidentiality (Maddox,
1984) and conflict of interest (sometimes financial; sometimes
scientific competition). Careful analyses of the review process also
showed a very poor inter-referee reliability (Zuckerman and Merton,
1971; Cole et al., 1981) - indeed little greater than chance (Gordon,
1977; Inglefinger, 1974; Rothwell and Martyn, 2000) - a clear
association with the reviewers experience and age (younger reviewers
producing more thorough reviews, Nylenna et al., 1994), a significant
tendency to favour positive findings (Mahoney, 1977) and a proclivity
towards projects and findings in line with the referees own ideas or
their "knee-jerk" adherence to current theoretical dogma (Ernst et al.,
1992), a phenomenon labelled "confirmatory bias" (Mahoney, 1977).
The process is hampered by a further bias: the preference for "normal
science" (Crawford, 1998). Refereeing tends to favour straightforward,
uncontroversial, even prosaic science, over more venturesome and
speculative arguments (Allen and Grant, 1998).
Moreover, detecting plagiarism is either based on a suspicion or on a
systematic review of all submissions to a journal - a trying and
lengthy process we rarely bother undertaking (Manweel and Baker, 1982;
Marshall, 1998). Time is a crucial factor. The mean time spent on
reviewing a manuscript is 1.5 hours (Lock and Smith, 1990; Nylenna et
al., 1994) with an average of one manuscript per month (Yankauer,
1990). Why should reviewers put more effort into a process which
carries no fame, no money, very little extra knowledge and for which
they bear no responsibility? We all work for the benefit of the
publishers; but in all other instances of this slavish subjugation
(publishing papers, reviewing books) our exacting ego is rewarded,
which is hardly the case in the anonymity of the refereeing process.
The problem is far from new (Huxley, 1901) and it has been discussed
before (Wilson, 1978; Peters and Ceci, 1980; Lock, 1985), though not so
much within Neuropsychology. The result of the reviewing process is
unsatisfying: referees are slow, hurried, often inaccurate (Horton,
1998) and sometimes cantankerous. Yet the process warrants good
science, no matter the means of publication (paper or web) or whether
the manuscripts are meant for restricted audience or free access.
We are sure that we could all amuse one another with endless anecdotes
on errors and misjudgements by referees and editors. Some experimental
evidence of the unfairness of peer-review comes from the provocative
study of Peter and Ceci (1982). They selected 12 psychology articles by
prestigious investigators and institutions and re-submitted them
(changing the names of the authors and using fictitious affiliations)
to the same 12 top American journals which had originally published
them some two years before. Three of these re-submissions were detected
as spoofs. Of the remaining nine, eight were rejected, not because of a
feeling of dej vu (lack of originality was never mentioned), but on the
basis of one or another major flaw in the study design. Surely a matter
for more than mere amusement.
Furthermore, in one journal or another, or even on the Internet, almost
everything is eventually published. Indeed, the few studies available
show that over 90% of the papers rejected by one journal are
eventually, and unaltered, published, not necessarily in a lower status
journal. The reviewing system appears to influence where but not what
should be published (Wilson, 1978): the mountain labours brings forth
... an undignified shuffling of manuscripts. One attempt to blatantly
and overtly deal with this phenomena is to create a central repository
of articles with cursory reviews thus placing the burden of deciding
whether a paper or study is important to the consumer Let the buyer
beware!
This musing is to open a debate and to make a (preliminary) proposal.
We are convinced that peer-review is central to scientific credibility.
However, as it stands the process is far from watertight. Is there any
way we can improve it by suggesting any modification, either radical or
minimal? Time is ripe for such a discussion to be launched (see the
JAMA and BMJ four congresses on peer review in biomedical publication:
www.jama-peer.org).
One possible alternative is to substitute referees with sponsors,
chosen by the authors, who overtly review and promote the papers (with
their names as sponsors on it) they regard worthy of publication prior
to submission. The manuscript would therefore be submitted to the
journal ready to be published, although the editorial board could still
reject it on scientific or non-scientific grounds (e.g.,
inappropriateness for the journal audience). On the one hand this will
make the refereeing process overt and rewarding for the reviewers. On
the other hand, it will require more responsibility, because the
sponsors names would be on it. Some problems are however immediately
apparent with this kind of review process: nepotism, favouritism
towards better known authors, over-commitment of the most popular
sponsors, awkwardness in requesting the review from sponsors, and, more
dangerously, scientific or personal covert blackmailing. All true,
though not dissimilar from the imperfections of the current system.
Would this proposal, all things considered, improve the present
cumbersome and not indisputable process? If not, does anybody have a
better idea? Is it even worth thinking about possible modifications of
the present system?
References
Allen L, Grant J. Funding innovations. Wellcome News, 1998, 16: 24-25.
Bakewell D. French research - publish in English or perish. Nature, 1992,
356:648.
Calza L, Garbisa S. Italian Professorships. Nature, 1995, 374: 492.
Cole S, Cole JR, Simon GA. Chance and consensus in peer-review. Science,
1981, 214: 881-886.
Crawford E. Nobel: Always the winners, never the losers. Science, 1998, 282:
1256-1257.
Ernst E, Resch KL, Uher EM. Reviewer bias. Annals of Internal
Medicine, 1992. 116: 958.
Forsdyke DR. On giraffes and peer-review. FASEB Journal, 1993. 7:619-621.
Gordon M. Evaluating the evaluators. New Scientist, 1977. 73: 342-343,
Horton R. Disseminating systematic reviews:working with medical journals.
Paper presented at the Symposium "Systematic Reviews: Beyond the Basic",
St.John College, Oxford, 8-9 January, 1998.
Huxley L. Life and letters of Thomas Henry Huxley. New York: D. Appleton and
Co.,1901, p.105.
Inglefinger FJ. Peer review in biomedical publication. American Journal of
Medicine, 1974, 56: 686-692,
Lock S. A difficult balance. Editorial peer review in Medicine. The Nuffield
Provincial Hospital Trust, 1985.
Lock S, Smith J. What do peer-reviewers do? Journal of the American
Medical Association, 1990, 263:1341-1343.
Maddox J. Privacy and the peer-review system. Nature, 1984. 312: 497.
Mahoney MJ. Publication prejudices: An experimental study of confirmatory
bias in the peer review system. Cognitive Therapy and Research, 1977. 1:
161-175,
Manwell C, Baker MA. Reform peer review. The Behavioral and Brain Science,
1982, 5: 211-225.
Marshall E. The Internet: A powerful tool for plagiarism sleuths. Science,
1998, 279: 474.
Nylenna M, Riis P, Karlsson Y. Multiple blinded reviews of the two
manuscripts - effects of referee characteristics and publication language.
Journal of the American Medical Association, 1994, 272: 149-151.
Perez-Enciso M. Spanish Practices. Nature, 1995, 378: 760.
Peters DP, Ceci, SJ. A manuscript masquerade. How well does the review
process work? The Sciences, 1980, 20: 16-19.
Peters DP, Ceci SJ. Peer-review practices of psychological journals: The
fate of published articles, submitted again. The Behavioral and Brain
Science, 1982. 5: 187-195. [See also the Commentary pp. 196-255].
Rothwell PM, Martyn CN. Reproducibility of peer review in clinical
neuroscience. Is agreement between reviewers any greater than would be
expected by chance alone? Brain, 123: 1964-1969.
Wallston Strudler B, O¹Leary VE. Sex makes a difference: differential
perceptions of women and men. Review of Personality and Social Psychology,
1982, 2: 9-43.
Wennerå s C, Wold A. Nepotism and sexism in peer-review. Nature, 1997, 387:
341-343.
Wilson JD. Peer review and publication. Journal of Clinical Investigation,
1978, 58: 1697-1701.
Yankauer A. Who are the peer reviewers and how much do they review?
Journal of the American Medical Association, 1990, 263: 1338-1340.
Received on Sat Feb 23 2002 - 16:56:04 GMT