The following article is excellent and accurate overall.
Cliffe, Rebeca (2006) Research Assessment Exercise: Bowing
out in Favour of Metrics. EPS Insights: 3 April 2006
http://www.epsltd.com/clients/viewUpdateNotes.asp?updateNoteID=1908
One can hardly quarrel with the following face-valid summary from
this article:
> "The move to a new metrics based system [for RAE] will no doubt please
> those who see a role for institutional repositories in monitoring
> research quality. The online environment has thrown up new metrics,
> which could be used alongside traditional measures such as citations.
> Usage can be measured at the point of consumption -- the number
> of "hits" on a particular article can indicate the uptake of the
> research. Web usage would be expected to be an early indicator of
> how often the article is later cited. Some believe that institutional
> repositories should be used as the basis for ongoing assessment of all
> UK peer-reviewed research output by mandating that researchers should
> place material in repositories. They argue that this would allow
> usage to be measured earlier, through downloads of both pre-prints
> and post-prints. Of course, this course of action would also advance
> the cause of open access by making this research available free."
But there are a few points of detail on which this otherwise accurate
report could be made even more useful:
(1) peer review has already been done for published articles, so
the issue is not (i) peer review vs. metrics but (ii) peer review
plus metrics vs. peer review plus metrics plus "peer re-review"
(by the RAE panels). It is the re-review of already peer-reviewed
publications that is the wasteful practice that needs to be scrapped,
given that peer review has already been done, and that metrics are
already highly correlated with the RAE ranking outcome anyway.
(2) For the fields in which the current RAE outcome is not already
highly correlated with metrics, further work is needed; obviously
works other than peer-reviewed article or books (e.g., artwork,
multimedia) will have to be evaluated in other ways, but for
science, engineering, biomedicine, social science, and most fields
of humanities, books and articles are the form that research output
takes, and they will be amenable to the increasingly powerful and
diverse forms of metrics that are being devised and tested. (Many
will be tested in parallel with the 2008 RAE, which will still be
conducted the old, wasteful way; some of the metrics may also be
testable retrospectively against prior RAE outcomes.)
> "Proponents of a metrics based system point to studies that show
> how average citation frequencies of articles can closely predict
> the scores given by the RAE for departmental quality, even though
> the RAE does not currently count these."
True, but the highest metric correlate of the present RAE outcome
is reportedly prior research funding (0.98). Yet it would be a big
mistake to scrap all other metrics and base the RAE rank on just
prior funding. That would just generate a massive Matthew Effect and
essentially make top-sliced RAE funding redundant with direct competitive
research project funding (thereby essentially "bowing out" of the dual
RAE/RCUK funding system altogether, reducing it to just research project
funding). What is remarkable about the high correlation between citation
counts and RAE ranks (0.7 - 0.9), even though the correlation is not quite
as high as with prior funding (0.98), is that citations are not presently
counted in the RAE (whereas prior funding is)! Not only are citations a
more independent metric of research performance than prior funding, but
counting them directly -- along with the many other candidate metrics --
can enrich and diversify the RAE evaluation, rather than just make it
into a self-fulfilling prophecy.
> "However, metrics tend to work better for the sciences than the
> humanities. Whereas the citation of science research is seen as an
> indicator of the quality and impact of the research, in the humanities
> this is not the case. Humanities research is based around critical
> discourse and an author may be citing an article simply to disagree
> with its argument."
I don't think this is quite accurate. It might be true that humanities
research makes less explicit use of citation counts today than science
research. It might even be true that the correlation between citation
counts and research productivity and importance is lower in the
humanities than in the sciences (though I am not aware of studies to that
effect). And it may also be true that citation counts in the humanities
are less correlated with RAE rankings than they are in the sciences. But
the familiar canard about articles being cited, not because they are
valid but important, but in order to disagree with them, has too much
the flavour of the a-priori dismissiveness of citation analysis that we
hear in *all* disciplines from those who have not really investigated
it, or the evidence for/against it, but are simply expressing their own
personal prejudices on the subject.
Let's see the citation counts for humanities articles and books, and
their correlation with other performance indicators as well as RAE
rankings rather than dismissing them a-priori on the basis of anecdotes.
> "Also, an analysis of RAE 2001 submissions revealed that while some
> 90% of research outputs listed by British researchers in the fields
> of Physics and Chemistry were mapped by ISI data, in Law the figure
> was below 10%, according to Ian Diamond of the Economic and Social
> Research Council (ESRC) (Oxford Workshop on the use of Metrics in
> Research Assessment)."
I am not sure what "mapped by ISI data" means, but if it means that ISI does not
cover enough of the pertinent journals in Law, then the empirical question is:
what are the pertinent journals? can citation counts be derived from their online
versions, using the publishers' websites and/or subscribing institutions' online
versions? how well does this augmented citation count correlate with the ISI
subsample (<10%)? and how well do both correlate with RAE ranking? (Surely ISI
coverage should not be the determinant of whether or not a metric is valid.)
> "Ultimately, a combination of qualitative and quantitative indicators
> would seem to be the best approach."
What is a "qualitative indicator"? A peer judgment of quality? But
that quality judgment has already been made by the peer-reviewers of
the journal in which the article was published -- and every field has a
hierarchy of quality among journals that is known (and may even sometimes
be correlated with the journal's impact factor, if one compares like
with like in terms of subject matter). What is the point of repeating
the peer review exercise? And especially if here too it turns out to be
correlated with metrics? Is it?
> "While metrics are likely to be used to simplify the research
> assessment process, the merits of a qualitative element would be to
> ensure that over-reliance on quantitative factors does not unfairly
> discriminate against research which is of good quality but has not
> been cited as highly as other research due to factors such as its
> local impact."
Why not ask the panels first to make quality judgments on the journals in which
the papers were published, and then see whether those rankings correlate with the
author/article citation metrics? and whether they correlate with the RAE rankings
based on the present time-consuming qualitative re-evaluations? If the
correlations prove lower than in the other fields (even when augmented by prior
funding and other metrics) *then* there may be a case for special treatment of
the humanities. Otherwise, the special pleading on behalf of uncited research
sounds as anecdotal, arbitrary and ad hoc as the claim that high citations
in humanities betoken disagreement rather than usage and importance,
as in other fields.
> "Unless more appropriate metrics can be developed for the humanities,
> it would seem that an element of expert peer review must remain in
> whatever metrics based system emerges from the ashes of the RAE."
It has not yet been shown whether the same metrics that correlate highly
with RAE outcome in other fields (funding, citations) truly fail to
do so in the humanities. If they do fail to correlate sufficiently,
there are still many candidate metrics to try out (co-citations,
downloads, book citations, CiteRank, latency/longevity, exogamy/endogamy,
hubs/authorities, co-text, etc.) before having to fall back on repeating,
badly, the peer evaluation that should have already have been done,
properly, by the journals in which the research first appeared.
Stevan Harnad
> Research Assessment Exercise: http://www.rae.co.uk
> Economic and Social Research Council: http://www.eserc.ac.uk
>
> Open access: practical matters become the key focus, EPS Insights, 10
> March 2005
> http://www.epsltd.com/accessArticles.asp?articleType=1&updateNoteID=1538
>
> Citation Analysis in the Open Access World, imi, September 2004
> http://www.epsltd.com/accessArticles.asp?articleType=2&articleID=236&imiID=294
Received on Fri Apr 07 2006 - 00:34:48 BST