Re: Future UK RAEs to be Metrics-Based

From: <l.hurtado_at_ED.AC.UK>
Date: Mon, 18 Sep 2006 20:37:48 +0100

Well, I'm all for empirically-based views in these matters. So, if
Openheim or others have actually soundly based studies showing what
Stevan and Openheim claim, then that's to be noted. I'll have to see
the stuff when it's published. In the meanwhile, a couple of further
questions:
--Pardon me for being out of touch, perhaps, but more precisely what is
being measured? What does journal "citation counts" refer to?
Citation of journal articles? Or citation of various things in journal
articles (and why privilege this medium?)? Or . . . what?
--What does "correlation" between RAE results and "citation counts"
actually comprise?

Let me lay out further reasons for some skepticism. In my own field
(biblical studies/theology), I'd say most senior-level scholars
actually publish very infrequently in refereed journals. We do perhaps
more in earlier years, but as we get to senior levels we tend (a) to
get requests for papers for multi-author volumes, and (b) we devote
ourselves to projects that best issue in book-length publications. So,
if my own productivity and impact were assessed by how many journal
articles I've published in the last five years, I'd look poor (even
though . . . well, let's say that I rather suspect that wouldn't be the
way I'm perceived by peers in the field).
Or is the metric to comprise how many times I'm *cited* in journals?
If so, is there some proven correlation between a scholar's impact or
significance of publications in the field and how many times he happens
to be cited in this one genre of publication? I'm just a bit
suspicious of the assumptions, which I still suspect are drawn (all
quite innocently, but naively) from disciplines in which journal
publication is much more the main and significant venue for scholarly
publication.
And, as we all know, "empirical" studies depend entirely on the
assumptions that lie at their base. So their value is heavily framed
by the validity and adequacy of the governing assumptions. No
accusations, just concerns.
Larry Hurtado

Quoting Stevan Harnad <harnad_at_ECS.SOTON.AC.UK>:

> On Mon, 18 Sep 2006, Larry Hurtado wrote:
>
>> Stevan and I have exchanged views on the *feasibility* of a metrics
>> approach to assessing research strength in the Humanities, and he's
>> impressed me that something such *might well* be feasible *when/if*
>> certain as-yet untested and undeveloped things fall into place. I note,
>> e.g., in Stevan's addendum to Oppenheim's comment that a way of handling
>> book-based disciplines "has not yet been looked at", and that a number
>> of other matters are as yet "untested".
>
> Larry is quite right that the (rather obvious and straightforward)
> procedure of self-archiving books' metadata and cited references in
> order to derive a comprehensive book-citation index (which would
> of course include journal articles citing books, books citing books,
> and books citing journal articles) had not yet been implemented or
> tested.
>
> However, the way to go about it is quite clear, and awaits only OA
> self-archiving mandates (to which a mandate to self-archive one's book
> metadata and reference list should be added as a matter of course).
>
> But please recall that I am an evangelist for OA self-archiving, because
> I *know* it can be done, that it works, and that it confers substantial
> benefits in terms of research access, usage and impact.
>
> Insofar as metrics are concerned, I am not an evangelist, but merely an
> enthusiast: The evidence is there, almost as clearly as it is with the
> OA impact-advantage, that citation counts are strongly correlated with
> RAE rankings in every discipline so far tested. Larry seems to pass over
> evidence in his remark about the as yet incomplete book citation data
> (ISI has some, but they are only partial). But what does he have to say
> about the correlation between RAE rankings and *journal article citation
> counts* in the humanities (i.e., in the "book-based" disciplines)?
> Charles will, for example, soon be reporting strong correlations in
> Music. Even without having to wait for a book-impact index, it seems
> clear that there are as yet no reported empirical exceptions to the
> correlation between journal article citation metrics and RAE outcomes.
>
> (I hope Charles will reply directly, posting some references to his and
> others' studies.)
>
>> This being the case, it is certainly not so a priori to say that a
>> metrics approach is not now really feasible for some disciplines.
>
> Nothing a priori about it: A posteriori, every discipline so far tested
> has shown positive correlations between its journal citation counts and its
> RAE rankings, including several Humanities disciplines.
>
> The advantage of having one last profligate panel-based RAE in parallel
> with the metric one in 2008 is that not a stone will be left unturned.
> If there prove to be any disciplines having small or non-existent
> correlations with metrics, they can and should be evaluated otherwise.
> But let us not assume, a priori, that there will be any such
> disciplines.
>
>> I emphasize that my point is not a philosophical one, but strictly
>> whether as yet a worked out scheme for handling all Humanities
>> disciplines rightly is in place, or capable of being mounted without
>> some significant further developments, or even thought out adequately.
>
> It depends entirely on the size of the metric correlations with the
> present RAE rankings. Some disciplines may need some supplementary forms
> of (non-metric) evaluation if their correlations are too weak. That is an
> empirical question. Meanwhile, the metrics will also be growing in power
> and diversity.
>
>> That's not an antagonistic question, simply someone asking for the
>> basis for the evangelistic stance of Stevan and some others.
>
> I evangelize for OA self-archiving of research and merely advocate
> further development, testing and use of metrics in research performance
> assessment, in all disciplines, until/unless evidence appears that there
> are exceptions. So far, the objections I know of are all only in the
> form of a priori preconceptions and habits, not objective data.
>
> Stevan Harnad
>
>> > Charles Oppenheim has authorised me to post this on his behalf:
>> >
>> > "Research I have done indicates that the same correlations between
>> > RAE scores and citation counts already noted in the sciences and
>> > social sciences apply just as strongly (sometimes more strongly)
>> > in the humanities! But you are right, Richard, that metrics are
>> > PERCEIVED to be inappropriate for the humanities and a lot of
>> > educating is needed on this topic."
>



L. W. Hurtado, Professor of New Testament Language, Literature & Theology
Director of Postgraduate Studies
School of Divinity, New College
University of Edinburgh
Mound Place
Edinburgh, UK. EH1 2LX
Office Phone: (0)131 650 8920. FAX: (0)131 650 7952
Received on Mon Sep 18 2006 - 23:07:58 BST