Re: Peer Review Reform Hypothesis-Testing

From: Donald Klein <DonaldK737_at_aol.com>
Date: Sat, 23 Nov 2002 15:07:26 +0000

research evaluation in terms of citations and impact is one valuable
approach. it's a type of productivity analysis. however certain important
published observations lie fallow for many years because of their
originality.

another complementary research evaluative procedure would be pedigree
analysis in terms of number, quality and originality of productive
trainees.

a frequent biographical statement by scientists is that at some early
career point they had an apprentice relationship to a mentor who provided
some combination of role model, resources, critical approach, creative
bent, substantive knowledge, work habits, etc etc that paid off in
generating a productive independent scientist. such pedigrees are open
to reliable independent evaluation (that's testable). fostering productive
mentorship is a complementary goal.

the reliability of grant peer review is little known but is probably
dismal. peer review validity could be tested by using current methods but
also randomly (double blind) funding non-approved grants and measuring
outcomes -- blind to funding -- five and ten years later. its not likely
that our granting agencies will self evaluate for validity. when i was
senior scientific advisor to ADAMHA this suggestion was laughed down.

as an alternative, productive mentors can be objectively discerned and
granted substantial funds to support their supervised trainees and trainee
studies (and to support themselves for supervisory work), without the time
wasting and originality inhibiting formalisms of project grant production.

mentors' judgements are probably far superior to "grant peer reviewers"
who are often in competitive (ideological/theoretical more than economic)
relationships -- this conflict of interest is regularly ignore -- and
often are not really peers.

grantsmanship consists in producing the most innocuous of documents since
originality is always both dicey and contentious. exhortations to grant
reviewers to support originality regularly fail. perhaps using mentors
with a demonstrated ability for original work to support juniors might
do better. it's testable.

further continued productivity, personal and via trainees, can be
estimated from factual observations rather than unreliable estimates
of the value of future research. therefore such grants do not have to
be sinecures -- perhaps run for 10 years (renewable). institutions that
attract and support such scientists should be granted funds for central
scientific resources.

mentoring is not everybody's abilty or interest -- nor is it essential
to being a productive scientist -- but it's an underrecognized important
talent that could be a sparkplug for scientific growth

Donald F. Klein
Professor of Psychiatry
Columbia University
Received on Sat Nov 23 2002 - 15:07:26 GMT