Re: Future UK RAEs to be Metrics-Based

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Fri, 16 Jun 2006 14:48:29 +0100

Willard McCarty makes some very valid points, but I am not sure we are
talking about the same thing. He objects to mechanical, industrial-style
evaluation of scholarly work ("metrics") and I agree with him completely
on this: I hate them too, and I think they are inimical to true quality
and individuality, both personal and disciplinary.

But with the RAE we are talking about a mass evaluation exercise,
at the university/department level, for top-sliced funding -- not about
individual research proposal funding, employment, salary, promotion,
tenure or prizes.

And we are talking about a system that is already in place and clearly
dysfunctional in its present form, because it both consumes a grotesque
amount of time and money that could instead be used for doing research
*and* because it generates virtually the same outcome (the same rankings)
as simple metrics would have done anyway (as the parallel metrics exercise
in 2008 will again confirm).

So it is not a vote on my part in favour of industrial-style academic
evaluation to say that *if* we are bent on having the RAE at all, we
may as well do it by metrics rather than by the profligate means we are
using presently.

Perhaps the right decision is that the RAE should be scrapped
altogether. I am not sure. I have a feeling that a dual system of mostly
peer-reviewed research bid funding, but "smoothed" and supplemented
by a smaller, more global, department-level competitive top-sliced
supplement may be better than than just individual grant peer review.
I don't know. My points are conditional on the existence and retention
of the RAE, and concern the ergonomics of it -- as well, of course, as
its bearing upon Open Access Self-Archiving, which a metric RAE could
both foster and enhance.

About the question of discipline differences, I am happy to replace my
rhetorical question ("Surely...") with three literal questions: (1)
Are there disciplines with no countable performance indicators? (2)
Which disciplines? (3) And if there are such disciplines, is it the
case that there is no way to do bulk RAE-style comparative assessments
of their research output other than by human panel re-review -- or the
appearance of it -- of their individual writings (current) RAE-style?

Willard's posting does not address this question (about RAE-style bulk
assessment) but instead speaks against conventional metrics (publication
and citation counts) in the context of individual academic assessment
(about which I have said nothing, but would, if asked, say that metrics
are a good supplement, but no substitute for individual, direct, human,
expert scholarly assessment).

About Willard's suggestion that we should "allow government ministers
and their appointees to make policy" I am afraid I can't be quite as
complacent, though I agree that ours is to clarify, not to legislate
(and that's precisely what I thought I was doing!).

May I suggest, though, that inasmuch this is an OA policy list, we discuss these
questions only inasmuch as they pertain to OA policy and its relation
to RAE (bulk assessment) policy, rather than to digress into more general issues
of scholarly evaluation which would be more appropriate for other lists?

Stevan Harnad

On Fri, 16 Jun 2006, Willard McCarty wrote:

> In his reply to Larry Hurtado, Stevan Harnad said,
>
> >No discipline uses metrics systematically yet; moreover, many metrics
> >are still to be designed and tested. However, the only thing really
> >"metrics" means is the objective measurement of quantifiable performance
> >indicators. Surely all disciplines have measurable performance
> >indicators. Surely it is not true of any discipline that the only way,
> >or the best way, to assess all of its annual research output is by having
> >each piece individually re-reviewed after it has already been peer-reviewed
> >twice -- before execution, by a funding council's peer-reviewers as
> >a research proposal, and after execution, by a journal's referees as a
> >research publication.
>
> The problem I have with the above is that statements which might be
> interesting, even helpful if they were interrogative are uttered as
> declarative, punctuated by "surely", as if an obvious doctrinal truth
> were somehow, inexplicably being overlooked. Getting doctrine right
> even goes a step beyond making policy, which at least can be
> pragmatic. Let's be less theological about research. Let's allow
> government ministers and their appointees to make policy. Let's
> concern ourselves with clarifying, rather than obscuring, the questions.
>
> Among the questions are these, I think:
>
> (1) How do we determine (e.g. for the purposes of funding) the worth
> of an individual's contributions to research?
> (2) How do we make sure that academic research, properly so called,
> continues in our increasingly market-driven, managerialized
> institutions of higher education?
>
> If intelligently pressed, (1) immediately generates the question of
> what we mean by "research". I don't intend here to be excessively
> abstract, rather to suggest that, as Tony Becher and Paul Trowler
> have said, "the ways in which particular groups of academics organize
> their professional lives are related in important ways to the
> intellectual tasks on which they are engaged" (Academic Tribes and
> Territories, 2nd edn, p 23). Unless one wishes to argue or indirectly
> to assert that a discipline is illegitimate, the discipline's
> particular way of doing things first needs to be understood, then
> respected. As in all those HSBC adverts in the UK, being polite or
> intolerable means very different things in different cultures; if you
> want to be successful in business around the world, you need to be
> polite and avoid being intolerable in various other people's
> radically differing terms.
>
> Becher and Trowler refer to a central essay by Clifford Geertz, "The
> Way We Think Now" (in Local Knowledge, 2000/1983). But one can also
> point to important work by Karin Knorr Cetina, in Epistemic Cultures
> (1999); to observations on disciplinarity by Richard Rorty, in
> Philosophy and the Mirror of Nature (1980); and to the echoes of
> Rorty throughout Peter Galison's work on the history of the natural
> sciences. It isn't helpful simply to declare that "Surely all
> disciplines have..."; one must actually take a look, to discover, as
> Rorty says, what counts by the criteria of a discipline's "normal
> discourse", i.e. everything "conducted within an agreed-upon set of
> conventions about what counts as a relevant contribution, what counts
> as answering a question, what counts as having a good argument for
> that answer or a good criticism of it" (1980: 320).
>
> Citation-counting may work in Kuhnian "normal science", to get a
> handle on the production-line productivity of those employed to grind
> out results, but I suspect that it's real home is in industrial
> models of research. I'd think it would tend to work against more
> creative, potentially revolutionary research. Perhaps in the
> sciences, as a rough and general rule, those who publish more and are
> cited more tend to be those we most value, but I'd hope that this and
> other like suppositions would be subjected to close scrutiny. I'd
> hope that we would take seriously examples such as Ludwik Fleck's or
> statements such as Einstein's, that "the temple [of science] would
> never have come to be, any more than a forest can grow which consists
> of nothing but creepers" ("Motive der Forschung", in Mein Weltbild),
> if the great oddballs were driven away by requirements that they
> measure up to some bureaucratic metric. A more recent Nobel laureate,
> John Polanyi, has noted in "Making Sense of Our Times" (1993) that
> great damage is done to basic research by imposing on it the
> standards of industrial manufacture. Somewhere he remarked that if
> the current system of rewards had been in place when he did his work,
> he'd not have come close to a Nobel Prize.
>
> In the humanities more or less the same obtains. Northrop Frye (my
> teacher) used to remark that if the rules of publication for tenure
> had been in force when he was a junior academic, he would never have
> survived: it took him something like 15 years to publish his first
> book. (As Hurtado remarked, books are what count in the humanities,
> and they take time.) In my own case, my first substantial book in the
> field I have helped to pioneer, humanities computing, took me close
> to 20 years, 10 before I wrote anything recognizable as part of it,
> the remaining 10 to understand the scope of the undertaking, read
> enough and think enough to be able to write it during my first
> sabbatical year. So one must look not just to the discipline someone
> inhabits but also to the stage in a person's career, to the state of
> the (existing or potential) discipline, to the size of the undertaking.
>
> About the second question, how we make sure research continues in
> universities. For all its flaws, the RAE does build research into the
> budgets of UK institutions of higher education. For my field, which
> has no panel to represent it, the problem it causes is one of fit.
> Humanities computing is not library and information science, not
> computer science, not any of the disciplines of the humanities with
> which it is intimately involved. Again, what's required is a much
> better understanding of the individuality of our epistemic island
> cultures and a much more flexible way of encouraging their healthy growth.
>
> I cannot begin to make a real argument here -- the subject is too
> vast for that, too vast for this medium. Rather I make a plea for us
> to question with some humility rather than to proclaim ex cathedra,
> to look into how others different from ourselves do their work, so we
> may determine what matters to them in their terms.
>
> Yours,
> WM
>
>
>
> Dr Willard McCarty | Reader in Humanities Computing | Centre for
> Computing in the Humanities | King's College London | Kay House, 7
> Arundel Street | London WC2R 3DX | U.K. | +44 (0)20 7848-2784 fax:
> -2980 || willard.mccarty_at_kcl.ac.uk www.kcl.ac.uk/humanities/cch/wlm/
>
>
Received on Fri Jun 16 2006 - 15:17:22 BST