Re: Future UK RAEs to be Metrics-Based
David Goodman seems to have misunderstood the RAE, the metric RAE, and my point
about the metric RAE:
(1) The RAE assesses peer-reviewed, published journal articles.
(2) For years now UK authors and their institutions have been wasting
their time re-submitting -- and the RAE panels have been wasting their time
re-reviewing -- *already peer-reviewed, published articles*.
(3) They have been re-reviewing them in order to generate the RAE
competitive rankings, in proportion to which the authors' departments
get top-sliced funding and prestige.
(4) The rankings turn out to be strongly correlated with citation counts
(even though citations are not actually counted by the panels) in all fields
tested so far.
(5) Hence most of the time and effort of re-submission and re-review
by the panels has been a complete waste, since much the same rankings could have
been generated by just counting citations (and other metrics).
(6) Hence it has at last been wisely decided to abandon the re-review and just use
metrics.
(7) In its one last incarnation in 2008, there will be a parallel exercise: both the
panel re-review and the metrics.
(8) The panels never did, and never could, perform peer review rigorously for each paper
submitted, because there was neither the time nor the expertise: and besides, the papers
had already been peer-reviewed by the journals in which they were published.
(9) Hence it was always absurd to insist that the submissions should
be the page images of the journal: Most only got skimmed anyway, and
the journal's quality level most likely exerts more influence on the
outcome than the peer expertise of the panel members to which each
paper happened to be assigned: this too was probably reflected in the
correlations with citations.
David Goodman seems to think that the RAE panel review was actually
performing the primary peer review -- refereeing for originality, primacy
correctness, etc. But that was already done, once (and once is enough)
by the journal's peer reviewers. That's not what the RAE panelists were
doing. (It's actually not clear *what* the RAE panelists were doing:
probably just assigning an RAE rank to each paper, based on one does
not quite know what.)
On Mon, 18 Sep 2006, David Goodman wrote:
> > SH:
> > (b) So what if authors correct or improve their publications after
> > publication? That's *good*, not bad!
>
> Stevan, you have previously argued consistently on this list and
> elsewhere that the formally published version is the version of record,
> not a version--improved or otherwise, that the author posted, previously
> or subsequently.
Yes. But what is your point? We are not talking here about the RAE
(or the author's IR) being a Deposit Library, archiving the version of
record. We are talking about a national research assessment exercise in
which a system that wasted everyone's time and money needlessly re-doing
peer-review (inexpertly) is to be replaced by metrics.
There was never any earthly reason why the RAE should have insisted on
seeing the publisher's PDF for its panel review -- and even less so now,
when the panel review is at long last fading out. The author's postprint
is and always was sufficient.
> If an author publishes something, and another person does the work
> better or differently, surely the original author cannot update his work
> after the fact and pretend he got it right the first time. He can, of
> course, publish an additional paper based on the corrected understanding.
Priority is examined and assigned by the original journal's
peer-reviewers, not by post-hoc RAE review panels assessing research
performance based on already published articles. It is not clear
what incentive there would be in fraudulently claiming priority by
doctoring a published article, but the risk -- for both the author
and his institutions -- of having the fraud discovered and made public
surely outweighs whatever imaginary incentive it might have had to do
it in the first place. Far more likely in an author's postprint is
a lapse in incorporating all the recommendations of the referees --
a lapse that puts the author at some slight risk of looking a bit worse
in a spot-check by the panelists. Correcting an error that slipped into
the published version, on the other hand, gives the author a slight
(and legitimate) chance of looking a bit better.
All trivial either way, since the panel review is being phased out for metrics
and was never rigorous enough to detect much of anything in the first place.
> If there is to be an RAE, should it not assess what the author
> published? Views like yours will give well-justified concern to
> administrators that the OA archives can not be trusted. I very much
> regret you have thought this way, let alone publicised it. It brings
> disrepute on us all.
David, I think you're getting a bit carried away...
The author's peer-reviewed final draft is fine for the RAE, and it's
also fine for an OA supplement in the author's IR, for users who cannot
afford to access the publisher's version of record. It is not a rival
publishing system, nor does it provide a rival version of record.
> > Charles Oppenheim:
> > "Research I have done indicates that the same correlations between
> > RAE scores and citation counts already noted in the sciences and
> > social sciences apply just as strongly (sometimes more strongly)
> > in the humanities! But you are right, Richard, that metrics are
> > PERCEIVED to be inappropriate for the humanities and a lot of
> > educating is needed on this topic."
>
> this would seem to imply that research in the humanities is now mainly
> dependent upon journal articles, as it is in the sciences.
No, it just implies that the RAE outcomes are correlated with article
citation counts, even in the humanities.
> -- and if its from the RAE, this might be applicable only in the UK?
Since only the UK does the RAE (Australia will soon have one too), and since the
correlation is between citations and RAE ranks, it clearly applies only to the UK.
(But one would expect similar effects for any similar assessment exercise.)
> (Might there be more difference between countries in the structure of
> humanities research than there is in the sciences.)
There might; or there might not. This is (yet another) empirical question that cannot be
answered from the speculative armchair.
> According to what I've seen in Chronicle
> of Higher Education and elsewhere, in US research universities the
> typical publication requirement for tenure is two monographs, and only
> a few such universities have been experimenting with accepting journal
> articles as a partial substitute.)
And what is your point here? The RAE is not evaluating individuals for tenure, it is
evaluating research performance of departments, for top-sliced research funding.
Stevan Harnad
Received on Tue Sep 19 2006 - 02:04:39 BST
This archive was generated by hypermail 2.3.0
: Fri Dec 10 2010 - 19:48:30 GMT