** Cross-Posted **
UK Research Assessment Reform Should Be Evidence-Based
Stevan Harnad
Hyperlinked version:
http://openaccess.eprints.org/index.php?/archives/307-guid.html
The UK Research Assessment Exercise has taken a few steps forward and a
few steps back:
http://www.thes.co.uk/current_edition/story.aspx?story_id=2038655
(1) In evaluating and rewarding the research performance of
universities department by department, future RAEs (after 2008) will
no longer, as before, assess only 4 selected papers per researcher,
among those researchers selected for inclusion: All papers, by all
departmental researchers, will be assessed. (Step forward)
(2) The assessment will be in terms of objective metrics, not just
in terms of panel review. (Step forward)
(3) The metrics will be multiple, rather than just a single metric.
(Step forward)
(4) The new system will apply to science, technology, and engineering,
at least. (Step forward)
(5) The new system may only apply to science, technology and
engineering. (Step Backward)
(6) The metrics considered may be only three, picked a priori:
(i) prior research income, (ii) postgraduate numbers, and (iii)
the "impact factor" (i.e., the average number of citations) of the
journal in which each article was published. (Step Backward)
As I have pointed out many times before, (i) prior research income, if
given too much weight, becomes a self-fulfilling prophecy, and reduces
the RAE to a multiplication factor on competitive research funding. The
result would be that instead of the current two autonomous components in
the UK's Dual Support System (RAE and RCUK), there would only be one:
RCUK (and other) competitive proposal funding, multiplied by the RAE
metric rank, dominated by prior funding.
To counterbalance against this, a rich spectrum of potential metrics
needs to be tested in the 2008 RAE, and validated against the panel
review rankings, which will still be collected in the 2008 parallel RAE.
Besides (i) research income, (ii) postgraduate student counts, and (iii)
journal impact factors, there is a vast spectrum of other candidate
metrics, including (iv) citation metrics for each article itself (rather
than just its journal's average), (iv) download metrics, (v) citation
and download growth curve metrics, (vi) co-citation metrics, (vii)
hub/authority metrics, (viii) endogamy/interdisciplinarity metrics (ix)
book citation metrics, (x) web link metrics, (xi) comment tag metrics,
(xii) course-pack metrics, and many more.
All these candidate metrics should be tested and validated against the
panel rankings in RAE 2008, in a multiple regression equation. The
selection and weighting of each metric should be adjusted, discipline by
discipline, rationally and empirically, rather than a priori, as is
being proposed now.
Harnad, S. (2007) Open Access Scientometrics and the UK Research
Assessment Exercise. In Proceedings of 11th Annual Meeting of the
International Society for Scientometrics and Informetrics 11(1), pp.
27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.
http://eprints.ecs.soton.ac.uk/13804/
(I might add that RCUK's plans to include "potential economic benefits
to the UK" among the criteria for competitive research funding could do
with a little more rational and empirical support too, rather than being
adopted a prior.)
http://www.thes.co.uk/current_edition/story.aspx?story_id=2038654
Stevan Harnad
Received on Fri Oct 12 2007 - 17:59:49 BST