Re: UK Research Assessment Exercise (RAE) review

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Thu, 21 Nov 2002 20:22:59 +0000

On Thu, 21 Nov 2002, David Goodman wrote:

> I question whether the members of any scientific field are qualified for
> judging quality in other scientific fields, except by the use of common
> sense and of objective measures, such as scientometric ones.

Agreed.

> I think librarians and other information science specialists are at
> least as qualified in both these aspects as others are.

Agreed. (For judging other disciplines. I expect that setting up a
national pan-disciplinary research assessment exercise probably draws on
a number of different lines of expertise, some of it having to do with
research methodology, some with statistics, some with research funding,
perhaps some with history and sociology of science and scholarship.)

> I further wonder whether the members of any scientific field are not in
> practice disqualified for evaluating departments in their own field by
> the inevitable effects of the old boy network.

Maybe not disqualified, but should perhaps have their numbers
counterbalanced by disinterested but knowledgeable parties.

> Not that this should disprove the argument, but I will mention that the
> proposal to evaluate the total scientific ouput of a group, good or bad,
> rather than just the best, will be eagerly supported by the publishers
> of the second-rate journals in which the lesser work appears.

Good point. But that's another reason why the quality-level of the journal
should be entered into the regression equation too. Salami-slicing should
have a scientometric signature too, which scientometric analysis should
be able to detect and weight accordingly.

> The particular improvement which is necessary is a way of measuring the
> influence not on the next years' papers, but on the next generations'.
> Thus I question the use of the current measurements for evaluating
> immediate research productivity for evaluating the actual value of
> the research.

And your alternative contender is...?

Once we have a full-text open-access database, with citation links,
co-citation analyses, hit-rates, time-series data, even inverted co-text
analyses, the predictive index could turn up as something as abstract
as the 2nd derivative or the latency to peak of the citation or the hit
growth curve. Unless you are suggesting that the only way to predict is
to retrodict (in which case the research assessment exercise's outcome
may come rather too late to reward the winning researcher...).

Stevan Harnad
Received on Thu Nov 21 2002 - 20:22:59 GMT