Re: Australia's RQF

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Sat, 18 Nov 2006 01:32:17 +0000 (GMT)

Linda Butler's follow-up posting is very sensible: metrics, but under
the scrutiny, for the time being, of panels. Australia is indeed being
both independent and innovative here. (And of course everything Linda
describes is perfectly congruent with what Arthur Sale described. The
niggles were about the weasel-words "quality" and "impact" -- both
vague, and slippery, with nothing much hanging on the distinction
and examples, as formulated!)

Stevan Harnad

On Sat, 18 Nov 2006, Linda Butler wrote:

> I think it is important to take account of what "phase", to use
> Stevan's term, each country is at.
>
> The UK is coming off several cycles of a traditional peer-review RAE,
> and is rightly questioning the cost of continuing with such an
> intensive process. It is hard to see the justification for
> continuing with the same system.
>
> HOWEVER, the UK does have the experience of several iterations of the
> RAE, and if the move to metrics throws up any anomalies, they will be
> relatively easy to detect.
>
> In contrast, Australia has had around 12 years of funding on the
> basis of a blunt formula that has little (I'm being generous here) to
> do with the quality of research. We don't have the UK's extensive
> knowledge set on the relative strengths and weaknesses of our
> departments.
>
> Some metrics may correlate well in some disciplines (maybe even
> many), but the correlation is rarely perfect. 'Quality' is a complex
> notion, and I don't share others' confidence that we can do away with
> the peer review element entirely. However, that does not mean the
> time-consuming assessment of a huge bundle of research outputs. What
> it does mean is that you still need a panel of experts to examine and
> interpret the metrics.
>
> For some panels, it might only take 5 minutes - for example, it's
> hard to see a bibliometric analysis of astronomy throwing up any
> surprises.
>
> But for others there might need to be considerably more effort. I'm
> currently working on an analysis of the 2002 RAE results in Political
> Science. If anyone can come up with any metric that would give War
> Studies at Kings College London a 5* rating I would be genuinely
> interested to hear.
>
> Australia is not moving "holus-bolus" to a UK RAE-clone. We can't
> afford it. We need metrics to help lighten the load on panels. But,
> at least for the first exercise, we need more than that.
>
> Add to that the fact that Australia is the first to systematically
> attempt to assess the "impact" (i.e. outside academia) of its
> research, and what is being proposed is hugely innovative. Lots of
> research councils, funding bodies, and others around the world are
> keen to see how this pans out - that is what they are increasingly
> seeking to do. And in Australia it is essential. There is a strong
> belief that merely demonstrating the quality of our research will not
> bring extra funds from government, as it has done in the UK. Rather
> what our government is looking for is a demonstration of the impact
> of that research in the wider community. It happens in spades - it's
> just that researchers/universities/etc have not been particularly
> successful at demonstrating the extent of it. Hopefully the RQF may
> go some way to addressing that and we may eventually get more
> research funding - we desperately need it.
>
> In the meantime, much work is being done in Australia to develop new
> metrics in those disciplines where the standard ones are not
> appropriate, and we have the strong support of researchers in these
> disciplines. So watch this space - Australia is not behind the game
> - it is ahead of it. Doesn't harm to be a bit provocative!!!
>
> Linda Butler
> Research Evaluation and Policy Project
Received on Sat Nov 18 2006 - 01:39:53 GMT