Re: Future UK RAEs to be Metrics-Based

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Tue, 19 Sep 2006 16:09:15 +0100

On Tue, 19 Sep 2006 l.hurtado_at_ED.AC.UK wrote:

> You're assuming that picking up items that happen to be cited in a
> selection of journals is somehow adequate, and it's THIS that I'm
> concerned about. What is your BASIS for your assumption about my field?

Charles can of course reply for himself, but I have to point out a systematic
dissociation here which does not seem to be converging:

Charles (and I) keep speaking about an empirical correlation between two sets of
empirical data: RAE rankings and citation counts.

Larry keeps talking about other things. We will not make progress this way.

The empirical fact seems to be that despite the difference in relative
emphasis on books and journals and despite the spotty coverage of
humanities citations so far, the correlation in the humanities disciplines
tested so far is nevertheless there, and big.

> Yes, of course, you can collect everything cited in a given set of
> journals, in principle. But is that the same thing as a representative
> picture of how scholarship is being treated in a field such as mine in
> which journals are not necessarily the principle medium in which
> scholarship is established or exhibited? This readily illustrates my
> concern about what assumptions go into experiments or "empirical"
> studies before they are run.

What assumptions are involved if I *observe* empirically that if I count
(currently available) citations in, say, Archaeology or Music or Library
Studies, and test their correlation with the corresponding RAE rankings
for Archaeology, Music, or Library Studies, then r = 0.8 (say),
meaning 64% of the variation in the RAE ranks can already be predicted
from the citation counts alone?

> So, QUESTION: What is your empirical basis for the assumption that
> simply monitoring a given set of journals is sufficient for any/all
> fields. You haven't addressed this yet.

That is not what Charles has said. He has said that there is a significant
and sizeable correlation between RAE outcome and the citation count metric
alone. That already speaks against continuing to do the time-wasting panel reviews
that currently generate the RAE outcomes; and there are more metrics than just
citation counts that can be used to approximate the panel outcome even more
closely.

The only kind of counter-evidence that is relevant to the question under discussion
is evidence of a small or nonexistent correlation between citation counts and RAE
ranks. So far, this outcome has not yet been observed for any discipline tested. It
may turn out to be the case for some disciplines, but so far there is no such
evidence.

Stevan Harnad

> Quoting "C.Oppenheim" <C.Oppenheim_at_LBORO.AC.UK>:
>
> > The question betrays a misunderstanding of how citation indexes work.
> > Citation indexes scan the journal literature for citations to all media, not
> > just other journals. So it makes no difference what vehicle the humanities
> > scholar disseminated his/her output in, the item will get picked up by the
> > citation index.
> >
> > The notion that a group of informed scholars could come up with a ranking
> > list over 30 minutes is an appealing one, but the fact remains that the UK's
> > RAE takes about a year to collect and analyse the data, together with many
> > meetings of the group of scholars, before decisions are made. one reason
> > for this tedious approach is to avoid legal challenges that the results were
> > not robustly reached.
>
Received on Tue Sep 19 2006 - 17:13:38 BST