For what it is worth, I am not personally a defender of the RAE. If
there is evidence that it merely entrenches Matthew Effects and other
inequities, it should be scrapped. My own welcoming of the switch to
metrics is a conditional one: (1) IF the RAE is to continue and (2)
IF the outcome is virtually the same with or without the wasteful
and time-consuming "peer-re-review" component, THEN the peer re-review
component should be scrapped in favour of just metrics.
I have, by the way, not heard one piece of objective evidence that metrics
would systematically disfavour anyone -- just a lot of (understandable,
but nonetheless objectively baseless) anecdotal anxiety that the
metrics-only RAE would disfavour one's own institution). Plus a lot of
(familiar, not-unique-to-RAE) superstition that if I am to be condemned
to the Tower (rightly or wrongly) let it be by a panel of my peers
(awake or asleep), not by a table of figures (even if the verdict is
exactly the same).
This might just have nothing to do with RAE or metrics at all, but
just human superstition (and irrationality) about base rate statistics
and their own unique, singular case:
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Explaining.Mind96/0221.html
http://en.wikipedia.org/wiki/Base_rate_fallacy
Stevan Harnad
On Fri, 7 Apr 2006, [Identity Deleted] wrote:
> It says something about metrics, that the precise figures of how many
> institutions, UoAs, panels, researchers submitted, etc don't really seem
> to have much of a bearing on one's view whether switching from the
> present system to the metrics system is a good or a bad thing. I guess
> we mostly look at it in terms of whether we expect our own departments
> to benefit or to lose out.
>
> Stevan suggested that it wasn't a capital issue and so criminal standards
> of proof were irrelevant. Well, for the department that I'm in it is
> not a capital issue, because our last RAE rating did not give us any
> funding council income, and for the Southamptons of this world it isn't
> a capital issue because their scale of research operation is so huge and
> their position is impregnable. But for middle-ranking institutions a
> shift from panel to metrics could conceivably (and according to THES
> will) lead to a massive cut in income, truly a death sentence for their
> research environment. And I'm not sure, either, how that sits with
> universities' commitment to shift research staff onto permanent
> contracts, something that has been an issue at least since the 70s.
>
> What puzzles me is this: if we hold that research is necessary to
> underpin teaching, why isn't research income based on student numbers -
> in effect why don't we say "if you want these graduates, then part of
> the cost of producing them is sustaining this research environment"? As
> Lord Melbourne said in praise of the House of Lords, "No damned nonsense
> about merit".
>
>
> -----Original Message-----
> Subject: Re: Future UK RAEs to be Metrics-Based
>
> From my observations on RAE 2001 i.e. anecdotal evidence:
>
> (a) Very far from all 170 institutions are submitting in all 70 UoAs:
> - an average is probably around 20-25 UoA per institution
> - in Computer Science area there are definitely less than 170
> institutions
>
> (b) An average submission of 10 researchers is probably too high too as
> I
> know a few Departments in Computer Science and Electronic Engineering
> which submitted between 2 and 8.
>
> Stevan Harnad wrote:
> > On Thu, 6 Apr 2006, "J.F.B.Rowland" wrote:
> >
> >> Date: Thu, 6 Apr 2006 16:41:57 +0100
> >>
> >>> SH: "There is no sense in repeating, with a local, inexpert UK
> panel,
> >>> what has already been done by each individual journal by
> >>> purpose-picked,
> >>> qualified experts."
> >>
> >> There is also no sense in gratuitous insults. The RAE panels are
> chosen
> >> from the most senior, experienced and expert academics in their
> >> respective
> >> fields in the UK. While one might agree that using such people for
> this
> >> purpose is unduly expensive and extravagant, they are well qualified
> to
> >> peer-review the work of others in their discipline.
> >
> > I yield to no one in my sincere admiration for UK experience,
> expertise
> > and seniority, but please do the arithmetic:
> >
> > (data source: RAE 2001 http://www.hero.ac.uk/rae/ )
> > c. 70 units of assessment
> > c. 10 panel members per unit
> > c. 170 institutions
> > c. 10 researchers submitted per unit/institution (guesstimate)
> > 4 papers per researcher
> >
> > That means 70 panels of 10 UK experts each have to be the
> peer-reviewers
> > for 10 x 170 x 4 = 6800 papers, or about 680 papers per peer.
> >
> > What do you think is the probability that the journals that originally
> > peer-reviewed and published those 6800 papers would have adjudged
> > those 10 UK peers to have been the right peer-reviewers for those 6800
> > papers? (Note that the journal can draw on all the qualified peers on
> > the planet, for each of its subspecialities: the RAE must draw on the
> > same 10 for all 6800 papers.)
> >
> > And note also that the peer review *has already been done*, for these
> > are peer-reviewed, published papers; so it is not clear what the UK
> panel
> > peer-reviewers can -- or should, or do -- do with their respective
> quotas
> > of 680 already-peer-reviewed papers each.
> >
> > I think there is no insult whatsoever in commending the RAE for
> abandoning
> > at last this gratuitous waste in time, money and expertise in favour
> of
> > the metrics that generate almost exactly the same outcome at a
> fraction
> > of the cost.
> >
> > Stevan Harnad
Received on Fri Apr 07 2006 - 11:42:15 BST