Re: Future UK RAEs to be Metrics-Based

From: <l.hurtado_at_ED.AC.UK>
Date: Wed, 14 Jun 2006 14:40:21 +0100

Stevan Harnad is totally in favour of a "metrics based" approach to
judging research merit with a view toward funding decisions, and greets
the news of such a shift from past/present RAE procedure with unalloyed
joy.

Well, hmmm. I'm not so sure (at least not yet). Perhaps there is more
immediate reason for such joy in those disciplines that already rely
heavily on a metrics approach to making decisions about researchers.
In the sciences, and also now social sciences, there are
citation-services that count publications and citations thereof in a
given list of journals deemed the "canon" of publication venues for a
given discipline. And in these disicplines journal articles are deemed
the main (perhaps sole) mode of research publication. Ok. Maybe it'll
work for these chaps.

But I'd like to know how it will work in Humanities fields such as
mine. Some questions, for Stevan or whomever. First, to my knowledge,
there is no such citation-count service in place. So, will the govt
now fund one to be set up for us? Or how will the metrics be compiled
for us? I.e., there simply is no mechanism in place for doing
"metrics" for Humanities disciplines.

Second, for us, journal articles are only one, and usually not deemed
the primary/preferred, mode of research publication. Books still count
quite heavily. So, if we want to count citations, will some
to-be-imagined citation-counting service/agency comb through all the
books in my field as well as the journal articles to count how many of
my publications get cited and how often? If not, then the "metrics"
will be so heavily flawed as to be completing misleading and useless.

Third, in many sciences, esp. natural and medical sciences, research
simply can't be conducted without significant external funding. But in
many/most Humanities disciplines truly groundbreaking and highly
influential research continues to be done without much external
funding. (Moreover, no govt has yet seen fit to provide funding for
the Humanities constituency of researchers commensurate with that
available for Sciences. So, it's a good thing we don't have to depend
on such funding!) My point is that the "metrics" for the Humanities
will have to be quite a bit different in what is counted, at the very
least.

Fourth, I'm not convinced (again, not yet; but I'm open to persuasion)
that counting things = research quality and impact. Example: A number
of years ago, coming from a tenure meeting at my previous University I
ran into a colleague in Sociology. He opined that it was unnecessary
to labor over tenure, and that he needed only two pieces of
information: number of publications and number of citations. I
responded, "I have two words for you: Pons and Flesichman". Remember
these guys? They were cited in Time and Newsweek and everywhere else
for a season as discovers of "cold fusion". And over the next couple
of years, as some 50 or so labs tried unsuccessfully to replicate their
alleged results, they must have been among the most frequently-cited
guys in the business. And the net effect of all that citation was to
discredit their work. So, citation = "impact". Well, maybe, but in
this case "impact" = negative impact. So, are we really so sure of
"metrics"?

Perhaps, however, Stevan can help me see the light, and join him in
acclaiming the advent of metrics.

L. W. Hurtado, Professor of New Testament Language, Literature & Theology
Director of Postgraduate Studies
School of Divinity, New College
University of Edinburgh
Mound Place
Edinburgh, UK. EH1 2LX
Office Phone: (0)131 650 8920. FAX: (0)131 650 7952
Received on Thu Jun 15 2006 - 22:21:33 BST