Re: UK Research Assessment Exercise (RAE) review

From: David Goodman <dgoodman_at_PHOENIX.PRINCETON.EDU>
Date: Fri, 22 Nov 2002 21:09:54 -0500

Yes, I do not think that the comparison of the eventual value of the
different specialties of scientific
research can be judged at the time the research is being done.
That requires historical knowledge as well as scientometrics.
This does imply a certain humility about the ability to use current
knowledge as a valid basis for long term science policy.
 The history of science offers abundant examples from all periods.

Your second derivative technique, if the data are sufficiently
accurate to support it, sounds like an exceeding nice way of measuring the
potential short-term rise of a scientific field (or department). I would
be reluctant to extrapolate very far into the future with such methods.

The quality of the work within a field
as done by different scientists is
another matter, and is a proper area for contemporaneous
measurement. It permits
the concentration of support on the apparently best work in all special
areas, without rejecting any sub-discipline's approach.

For example, as judged by apparent current productivity, and its apparent
valuation by the scientific world in general, scientometrics does not
show very well. You and I know better, of course. :)

 On
Thu, 21 Nov 2002, Stevan Harnad wrote:

> On Thu, 21 Nov 2002, David Goodman wrote:
>
> > I question whether the members of any scientific field are qualified for
> > judging quality in other scientific fields, except by the use of common
> > sense and of objective measures, such as scientometric ones.
>
> Agreed.
>
> > I think librarians and other information science specialists are at
> > least as qualified in both these aspects as others are.
>
> Agreed. (For judging other disciplines. I expect that setting up a
> national pan-disciplinary research assessment exercise probably draws on
> a number of different lines of expertise, some of it having to do with
> research methodology, some with statistics, some with research funding,
> perhaps some with history and sociology of science and scholarship.)
>
> > I further wonder whether the members of any scientific field are not in
> > practice disqualified for evaluating departments in their own field by
> > the inevitable effects of the old boy network.
>
> Maybe not disqualified, but should perhaps have their numbers
> counterbalanced by disinterested but knowledgeable parties.
>
> > Not that this should disprove the argument, but I will mention that the
> > proposal to evaluate the total scientific ouput of a group, good or bad,
> > rather than just the best, will be eagerly supported by the publishers
> > of the second-rate journals in which the lesser work appears.
>
> Good point. But that's another reason why the quality-level of the journal
> should be entered into the regression equation too. Salami-slicing should
> have a scientometric signature too, which scientometric analysis should
> be able to detect and weight accordingly.
>
> > The particular improvement which is necessary is a way of measuring the
> > influence not on the next years' papers, but on the next generations'.
> > Thus I question the use of the current measurements for evaluating
> > immediate research productivity for evaluating the actual value of
> > the research.
>
> And your alternative contender is...?
>
> Once we have a full-text open-access database, with citation links,
> co-citation analyses, hit-rates, time-series data, even inverted co-text
> analyses, the predictive index could turn up as something as abstract
> as the 2nd derivative or the latency to peak of the citation or the hit
> growth curve. Unless you are suggesting that the only way to predict is
> to retrodict (in which case the research assessment exercise's outcome
> may come rather too late to reward the winning researcher...).
>
> Stevan Harnad
>

Dr. David Goodman
Biological Sciences Bibliographer
Princeton University Library
dgoodman_at_princeton.edu
Received on Sat Nov 23 2002 - 02:09:54 GMT