Validating Multiple Metrics As Substitutes for Expert Evaluation of Research Performance

From: Stevan Harnad <amsciforum_at_GMAIL.COM>
Date: Tue, 13 Jan 2009 15:52:22 -0500

Nature's editorial "Experts still needed" (Nature 457: 7-8, 1 January
2009) is right that no one metric alone can substitute for the expert
evaluation of research performance (based on already-published,
peer-reviewed research), because no single metric (including citation
counts) is strongly enough correlated with expert judgments to take
their place. However, some individual metrics (such as citation
counts) are nevertheless significantly correlated with expert
judgments; and it is likely that a battery of multiple metrics, used
jointly, will be even more strongly correlated with expert judgments.
That is the unique opportunity that the current UK Research
Assessment Exercise (RAE) -- and our open, online age, with its rich
spectrum of potential performance indicators -- jointly provide: the
opportunity to systematically cross-validate a rich and diverse
battery of candidate metrics of research productivity, performance
and impact (including citations, co-citations, downloads, tags,
growth/decay metrics, etc.) against expert judgments, field by field.
The rich data that the 2008 RAE returns have provided make it
possible to do this validation exercise now too, for all disciplines,
on a major nation-sized database. If successfully validated, the
metric batteries can then not only pinch-hit for experts in future
RAEs, but they will provide an open database that allows anyone,
anywhere, any time to do comparative evaluations of research
performance: continuous assessment and answerability.

(Note that what is at issue is whether metrics can substitute for
costly and time-consuming expert rankings in the retrospective
assessment of published, peer-reviewed research. It is of course
not peer review itself -- another form of expert judgment -- that
metrics are being proposed to replace [or simplify and supplement],
for either submitted papers or research proposals.)

Harnad, S. (2008) Validating Research Performance Metrics Against
Peer Rankings. Ethics in Science and Environmental Politics 8 (11)
doi:10.3354/esep00088 Special Issue: The Use And Misuse Of
Bibliometric Indices In Evaluating Scholarly Performance 

Stevan Harnad
American Scientist Open Access Forum
Received on Tue Jan 13 2009 - 20:52:56 GMT