Re: Future UK RAEs to be Metrics-Based

From: Willard McCarty <willard.mccarty_at_KCL.AC.UK>
Date: Fri, 16 Jun 2006 10:39:52 +0100

In his reply to Larry Hurtado, Stevan Harnad said,

>No discipline uses metrics systematically yet; moreover, many metrics
>are still to be designed and tested. However, the only thing really
>"metrics" means is the objective measurement of quantifiable performance
>indicators. Surely all disciplines have measurable performance
>indicators. Surely it is not true of any discipline that the only way,
>or the best way, to assess all of its annual research output is by having
>each piece individually re-reviewed after it has already been peer-reviewed
>twice -- before execution, by a funding council's peer-reviewers as
>a research proposal, and after execution, by a journal's referees as a
>research publication.

The problem I have with the above is that statements which might be
interesting, even helpful if they were interrogative are uttered as
declarative, punctuated by "surely", as if an obvious doctrinal truth
were somehow, inexplicably being overlooked. Getting doctrine right
even goes a step beyond making policy, which at least can be
pragmatic. Let's be less theological about research. Let's allow
government ministers and their appointees to make policy. Let's
concern ourselves with clarifying, rather than obscuring, the questions.

Among the questions are these, I think:

(1) How do we determine (e.g. for the purposes of funding) the worth
of an individual's contributions to research?
(2) How do we make sure that academic research, properly so called,
continues in our increasingly market-driven, managerialized
institutions of higher education?

If intelligently pressed, (1) immediately generates the question of
what we mean by "research". I don't intend here to be excessively
abstract, rather to suggest that, as Tony Becher and Paul Trowler
have said, "the ways in which particular groups of academics organize
their professional lives are related in important ways to the
intellectual tasks on which they are engaged" (Academic Tribes and
Territories, 2nd edn, p 23). Unless one wishes to argue or indirectly
to assert that a discipline is illegitimate, the discipline's
particular way of doing things first needs to be understood, then
respected. As in all those HSBC adverts in the UK, being polite or
intolerable means very different things in different cultures; if you
want to be successful in business around the world, you need to be
polite and avoid being intolerable in various other people's
radically differing terms.

Becher and Trowler refer to a central essay by Clifford Geertz, "The
Way We Think Now" (in Local Knowledge, 2000/1983). But one can also
point to important work by Karin Knorr Cetina, in Epistemic Cultures
(1999); to observations on disciplinarity by Richard Rorty, in
Philosophy and the Mirror of Nature (1980); and to the echoes of
Rorty throughout Peter Galison's work on the history of the natural
sciences. It isn't helpful simply to declare that "Surely all
disciplines have..."; one must actually take a look, to discover, as
Rorty says, what counts by the criteria of a discipline's "normal
discourse", i.e. everything "conducted within an agreed-upon set of
conventions about what counts as a relevant contribution, what counts
as answering a question, what counts as having a good argument for
that answer or a good criticism of it" (1980: 320).

Citation-counting may work in Kuhnian "normal science", to get a
handle on the production-line productivity of those employed to grind
out results, but I suspect that it's real home is in industrial
models of research. I'd think it would tend to work against more
creative, potentially revolutionary research. Perhaps in the
sciences, as a rough and general rule, those who publish more and are
cited more tend to be those we most value, but I'd hope that this and
other like suppositions would be subjected to close scrutiny. I'd
hope that we would take seriously examples such as Ludwik Fleck's or
statements such as Einstein's, that "the temple [of science] would
never have come to be, any more than a forest can grow which consists
of nothing but creepers" ("Motive der Forschung", in Mein Weltbild),
if the great oddballs were driven away by requirements that they
measure up to some bureaucratic metric. A more recent Nobel laureate,
John Polanyi, has noted in "Making Sense of Our Times" (1993) that
great damage is done to basic research by imposing on it the
standards of industrial manufacture. Somewhere he remarked that if
the current system of rewards had been in place when he did his work,
he'd not have come close to a Nobel Prize.

In the humanities more or less the same obtains. Northrop Frye (my
teacher) used to remark that if the rules of publication for tenure
had been in force when he was a junior academic, he would never have
survived: it took him something like 15 years to publish his first
book. (As Hurtado remarked, books are what count in the humanities,
and they take time.) In my own case, my first substantial book in the
field I have helped to pioneer, humanities computing, took me close
to 20 years, 10 before I wrote anything recognizable as part of it,
the remaining 10 to understand the scope of the undertaking, read
enough and think enough to be able to write it during my first
sabbatical year. So one must look not just to the discipline someone
inhabits but also to the stage in a person's career, to the state of
the (existing or potential) discipline, to the size of the undertaking.

About the second question, how we make sure research continues in
universities. For all its flaws, the RAE does build research into the
budgets of UK institutions of higher education. For my field, which
has no panel to represent it, the problem it causes is one of fit.
Humanities computing is not library and information science, not
computer science, not any of the disciplines of the humanities with
which it is intimately involved. Again, what's required is a much
better understanding of the individuality of our epistemic island
cultures and a much more flexible way of encouraging their healthy growth.

I cannot begin to make a real argument here -- the subject is too
vast for that, too vast for this medium. Rather I make a plea for us
to question with some humility rather than to proclaim ex cathedra,
to look into how others different from ourselves do their work, so we
may determine what matters to them in their terms.

Yours,
WM



Dr Willard McCarty | Reader in Humanities Computing | Centre for
Computing in the Humanities | King's College London | Kay House, 7
Arundel Street | London WC2R 3DX | U.K. | +44 (0)20 7848-2784 fax:
-2980 || willard.mccarty_at_kcl.ac.uk www.kcl.ac.uk/humanities/cch/wlm/
Received on Fri Jun 16 2006 - 14:48:44 BST