On Fri, 16 Jun 2006, Larry Hurtado wrote:
> Ah, OK. To my concerns about the feasibility of a metrics-based
> approach to assessing research in the Humanities, Stevan's very
> informative response is that . . . well, a LOT more work (which = a LOT
> of financial resourcing of the work) will be needed.
No, I didn't exactly say that. I said existing standard metrics
(citations, funding, students) are already a good first cut for many
disciplines; and that an Open Access database containing the full texts of
all articles and at least the bibliographies of all books will generate
candidate metrics for all disciplines.
The next RAE is a "parallel" exercise, where submissions will be done the old,
profligate way, and in parallel, the standard metrics will be calculated, to compare
outcomes, for each discipline. It is an empirical question now, how well these two
ranking methods will agree, for each discipline, hence an empirical question how much
more work will be needed to get a valid metric ranking for each discipline.
I would add only that for book-based disciplines it is already possible to get a
1st-approximation book-impact metric, using ISI (which includes books citations) and
Google Scholar. An OA database would greatly enhance this metric.
> E.g., we'll all
> have to scan our publications (or the footnotes at least, hmmm, not
> quite sure how to do that) into digital form, and have them mounted on
> a univ web site. Well, I'd be happy with all this, but who will do it?
First, the scanning recommendation was an optional one, and was meant
for the legacy literature (i.e., books published *before* the current RAE
window). For current books (and even for recent ones) the bibliography of
works cited is presumably still available in digital form in the author's
word-processor; no need to scan anything. (Yes, it would be advisable,
in fields that use odd reference formats, to generate a simple list
of cited works in a simple, standard format (authors, date, title,
publication [journal name, book chapter reference, etc.], publisher)
and self-archive that. It's a trivial amount of work and truly not worth
worrying about here: it's not like scanning and OCR for full texts,
which, as I said, is just a suggested option for the legacy literature,
for enterprising authors...)
The digital reference list is prepared by the same person who writes the work: the
author, and it is trivial.
> Given the funding practices in UK universities, it will probably be
> me (yes, even UK professors . . . at least in the Humanities, don't
> have their own assistants and secretaries to do such work). So,
> instead of preparing RAE submission documents, which takes time away
> from research, I'll have to scan manually all my publications into some
> sort of digital format, and get them mounted on an appropraite site,
> which will = clerical work requiring time from research.
No. No scanning. And depositing a digital document in your institutional
repository is just a few minutes worth of keystrokes per publication,
which you, or your students, or your departal clerical help are quite
capable of performing, and which you will agree is trivial, if you simply
try it a few times (which is all it will take, per year!)
Carr, L. and Harnad, S. (2005) Keystroke Economy: A Study of the
Time and Effort Involved in Self-Archiving.
http://eprints.ecs.soton.ac.uk/10688/
> Granted, spending money on an IT setup instead of millions on running
> RAE committees might be a saving. Don't know. Given the recent
> histories of govt IT systems, however, I'm not so sure.
Trust me: It takes incomparably less time (and costs incomparably less)
to deposit all annual publications in one's Institutional Repository
than to do the RAE paper-chase. Try it and you'll see what I mean.
> So, essentially, Stevan's enthusiasm for a metrics approach is
> dependent on his vision of OA compliance by us all . . . scholars in
> all fields, publishers, universities, govts, funding bodies, etc. When
> that's all in place, let me know.
Larry is quite right to say that the RAE switch to metrics is not enough.
The RCUK needs to mandate self-archiving. Let's hope they will have the
sense to do so.
> Oh, and my reservations about a metrics approach don't = any particular
> great enthusiasm for the current RAE approach either. So, if Stevan
> could refrain from such non sequiturs, I'd be grateful. It's a nice
> debating trick, but we're not into such tricks here, I hope. We're
> trying to figure out a sane, economic and relatively efficient way of
> trying to make decisions about quality of research. Actually, my own
> provisional preference (at least till Stevan fully persuades me
> otherwise) is to consider simply requiring researchers to make periodic
> grant applications to the funding councils for operating grants, which
> effecitively means abolishing the two-envelope approach to
> INSTITUTIONAL funding, and restricting RESEARCH funding to researchers.
I don't think I was using a debating trick. My support for a metric RAE is based
entirely on the premise that there will continue to be an RAE at all. Under those
conditions, a metric RAE is incomparably preferable to the current profligate paper
chase and panel re-review. On whether the RAE should be scrapped altogether, nolo
contendere (although I have a mild intuitive preferance for a dual system, where
some small department-level top-slicing smooths and supplements individual grant
submissions). This Forum is about OA policy, however, not about research funding
policy, so the RAE is pertinent only inasmuch as it exists, and can help nudge the
RCUK to follow suit by doing the sensible thing.
And I too detest mass metrics as inimical to individual quality and creativity. But
I'm not happy with the gaussian decision-making that underpins democracy either. I
just can't think of any good alternative at those scales...
About the value of OA, however, I have no doubts. And inasmuch as rationalizing the
RAE system with metrics can help usher in OA, it seems sensible to support it.
Stevan Harnad
Received on Fri Jun 16 2006 - 16:00:08 BST