Re: UK Research Assessment Exercise (RAE) review

From: Jan Velterop <jan_at_BIOMEDCENTRAL.COM>
Date: Wed, 27 Nov 2002 10:08:15 -0000

The semantic whip "what is scientometrics?" may lash, but doesn't quite
crack, in my opinion. If Stevan says "I don't think that in reminding us
[...], Jan is not giving us an alternative to scientometric
quantification.", does that mean that he *does* think I *do*?

Good. I didn't even mean to.

I meant to give an example of a complement to quantification.

Much of the trouble is not quantification per se, but the lack of
information to enable weighting the votes. The journals (well, at least some
of them) lend a certain weight to their peer-review, but this peer-review is
almost always anonymous. Reviewers may not even be proper 'peers' in some
cases. Stevan speculates that "Perhaps reviewer-names could accrue some
objective scientometric weight...". I would perhaps remove the 'perhaps'.
Maybe it has its own set of problems, but disclosing the peers' identity may
be a great help in assessing the weight or significance of the review.
Besides, it may disclose possible conflicts of interest. All BMC's medical
journals have open peer review which works most satisfactorily. All journals
also have a comments section enabling a public, open discussion.

The point of Faculty of 1000 is that an open, secondary review of published
literature by acknowledged leaders in the field, signed by the reviewer, is
seen by increasing numbers of researchers (measured by the fast-growing
usage figures of F1000) as a very meaningful addition to quantitative data
and a way to sort and rank articles in order of importance. Of course one
can subsequently quantify such qualitative information. But what a known and
acknowledged authority thinks of an article is to many more interesting than
what anonymous peer-reviewers think. Any research assessment exercise should
seriously look at resources such as offered by Faculty of 1000.
Received on Wed Nov 27 2002 - 10:08:15 GMT