The Need to Cross-Validate and Initialize Multiple Metrics Jointly Against Peer Ratings

From: Stevan Harnad <amsciforum_at_GMAIL.COM>
Date: Thu, 19 Mar 2009 09:18:31 -0400

The Times Higher Education Supplement (THES) has reported the results
of a study they commissioned by Evidence Ltd that found that the
ranking criteria for assessing and rewarding research performance in
the UK Research Assessment Exercise (RAE) changed from RAE
2001 to RAE 2008. The result is that citations, which correlated
highly with RAE 2001, correlated less highly with RAE 2008, so a
number of universities whose citation counts had decreased were
rewarded more in 2008, and a number of universities whose citation
counts had increased were rewarded less.

(1) Citation counts are only one (though an important one) among many
potential metrics of research performance.

(2) If the RAE peer panel raters' criteria for ranking the
universities varied or were inconsistent between RAE 2001 and RAE
2008 then that is a problem with peer ratings rather than with
metrics (which, being objective, remain consistent).

(3) Despite the variability and inconsistency, peer ratings are the
only way to initialise the weights on metrics: Metrics first have to
be jointly validated against expert peer evaluation by measuring
their correlation with the peer rankings, discipline by discipline;
then the metrics' respective weights can be updated and fine-tuned,
discipline by discipline, in conjunction with expert judgment of the
resulting rankings and continuing research activity.

(4) If only one metric (e.g., citation) is used, there is the risk
that expert ratings will simply echo it. But if a rich and diverse
battery of multiple metrics is jointly validated and initialized
against the RAE 2008 expert ratings, then this will create an
assessment-assistant tool whose initial weights can be calibrated and
used in an exploratory way to generate different rankings, to be
compared by the peer panels with previous rankings as well as with
new, evolving criteria of research productivity, uptake, importance,
influence, excellence and impact.

(5) The dawning era of Open Access (free web access) to peer-reviewed
research is providing a wealth of new metrics to be included, tested
and assigned initial weights in the joint battery of metrics. These
include download counts, citation and download growth and decay
rates, hub and authority scores, interdisciplinarity scores,
co-citations, tag counts, comment counts, link counts, data-usage,
and many other openly accessible and measurable properties of the
growth of knowledge in our evolving "Cognitive Commons."

Brody, T., Kampa, S., Harnad, S., Carr, L. and Hitchcock, S.
(2003) Digitometric Services for Open Archives Environments.
In Proceedings of European Conference on Digital Libraries 2003, pp.
207-220, Trondheim, Norway. 

Brody, T., Carr, L., Harnad, S. and Swan, A. (2007) Time to Convert
to Metrics. Research Fortnight pp. 17-18. 

Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A.
(2007) Incentivizing the Open Access Research Web:
Publication-Archiving, Data-Archiving and Scientometrics. CTWatch
Quarterly 3(3). 

Carr, L., Hitchcock, S., Oppenheim, C., McDonald, J. W., Champion, T.
and Harnad, S. (2006) Extending journal-based research impact
assessment to book-based disciplines. Technical Report, ECS,
University of Southampton.

Harnad, S. (2001) Research access, impact and assessment. Times
Higher Education Supplement 1487: p. 16.

Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated
online RAE CVs Linked to University Eprint Archives: Improving the UK
Research Assessment Exercise whilst making it cheaper and easier.
Ariadne 35. 

Harnad, S. (2006) Online, Continuous, Metrics-Based Research
Assessment. Technical Report, ECS, University of Southampton. 

Harnad, S. (2007) Open Access Scientometrics and the UK Research
Assessment Exercise. In Proceedings of 11th Annual Meeting of the
International Society for Scientometrics and Informetrics 11(1), pp.
27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. 

Harnad, S. (2008) Self-Archiving, Metrics and Mandates. Science
Editor 31(2) 57-59

Harnad, S. (2008) Validating Research Performance Metrics Against
Peer Rankings. Ethics in Science and Environmental Politics 8 (11)
doi:10.3354/esep00088 The Use And Misuse Of Bibliometric Indices In
Evaluating Scholarly Performance 

Harnad, S., Carr, L. and Gingras, Y. (2008) Maximizing Research
Progress Through Open Access Mandates and Metrics. Liinc em Revista.

Harnad, S. (2009) Multiple metrics required to measure research
performance. Nature (Correspondence) 457 (785) (12 February 2009) 
Received on Thu Mar 19 2009 - 13:20:29 GMT