Re: Future UK RAEs to be Metrics-Based

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Mon, 18 Sep 2006 19:07:44 +0100

On Mon, 18 Sep 2006, Larry Hurtado wrote:

> Stevan and I have exchanged views on the *feasibility* of a metrics
> approach to assessing research strength in the Humanities, and he's
> impressed me that something such *might well* be feasible *when/if*
> certain as-yet untested and undeveloped things fall into place. I note,
> e.g., in Stevan's addendum to Oppenheim's comment that a way of handling
> book-based disciplines "has not yet been looked at", and that a number
> of other matters are as yet "untested".

Larry is quite right that the (rather obvious and straightforward)
procedure of self-archiving books' metadata and cited references in
order to derive a comprehensive book-citation index (which would
of course include journal articles citing books, books citing books,
and books citing journal articles) had not yet been implemented or
tested.

However, the way to go about it is quite clear, and awaits only OA
self-archiving mandates (to which a mandate to self-archive one's book
metadata and reference list should be added as a matter of course).

But please recall that I am an evangelist for OA self-archiving, because
I *know* it can be done, that it works, and that it confers substantial
benefits in terms of research access, usage and impact.

Insofar as metrics are concerned, I am not an evangelist, but merely an
enthusiast: The evidence is there, almost as clearly as it is with the
OA impact-advantage, that citation counts are strongly correlated with
RAE rankings in every discipline so far tested. Larry seems to pass over
evidence in his remark about the as yet incomplete book citation data
(ISI has some, but they are only partial). But what does he have to say
about the correlation between RAE rankings and *journal article citation
counts* in the humanities (i.e., in the "book-based" disciplines)?
Charles will, for example, soon be reporting strong correlations in
Music. Even without having to wait for a book-impact index, it seems
clear that there are as yet no reported empirical exceptions to the
correlation between journal article citation metrics and RAE outcomes.

(I hope Charles will reply directly, posting some references to his and
others' studies.)

> This being the case, it is certainly not so a priori to say that a
> metrics approach is not now really feasible for some disciplines.

Nothing a priori about it: A posteriori, every discipline so far tested
has shown positive correlations between its journal citation counts and its
RAE rankings, including several Humanities disciplines.

The advantage of having one last profligate panel-based RAE in parallel
with the metric one in 2008 is that not a stone will be left unturned.
If there prove to be any disciplines having small or non-existent
correlations with metrics, they can and should be evaluated otherwise.
But let us not assume, a priori, that there will be any such
disciplines.

> I emphasize that my point is not a philosophical one, but strictly
> whether as yet a worked out scheme for handling all Humanities
> disciplines rightly is in place, or capable of being mounted without
> some significant further developments, or even thought out adequately.

It depends entirely on the size of the metric correlations with the
present RAE rankings. Some disciplines may need some supplementary forms
of (non-metric) evaluation if their correlations are too weak. That is an
empirical question. Meanwhile, the metrics will also be growing in power
and diversity.

> That's not an antagonistic question, simply someone asking for the
> basis for the evangelistic stance of Stevan and some others.

I evangelize for OA self-archiving of research and merely advocate
further development, testing and use of metrics in research performance
assessment, in all disciplines, until/unless evidence appears that there
are exceptions. So far, the objections I know of are all only in the
form of a priori preconceptions and habits, not objective data.

Stevan Harnad

> > Charles Oppenheim has authorised me to post this on his behalf:
> >
> > "Research I have done indicates that the same correlations between
> > RAE scores and citation counts already noted in the sciences and
> > social sciences apply just as strongly (sometimes more strongly)
> > in the humanities! But you are right, Richard, that metrics are
> > PERCEIVED to be inappropriate for the humanities and a lot of
> > educating is needed on this topic."
Received on Mon Sep 18 2006 - 19:20:44 BST