On Sun, 15 Jun 2008, Loet Leydesdorff wrote:
>> SH: But what all this valuable, valid cautionary discussion overlooks is not
>> only the possibility but the *empirically demonstrated fact* that there
>> exist metrics that are highly correlated with human expert rankings.
>
> It seems to me that it is difficult to generalize from one setting in which
> human experts and certain ranks coincided to the *existence *of such
> correlations across the board. Much may depend on how the experts are
> selected. I did some research in which referee reports did not correlate
> with citation and publication measures.
Much may depend on how the experts are selected, but that was just as
true during the 20 years in which rankings by experts were the sole
criterion for the rankings in the UR Research Assessment Exercise
(RAE). (In validating predictive metrics one must not endeavor to be
Holier than the Pope: Your predictor can at best hope to be as good as,
but not better than, your criterion.)
That said: All correlations to date between total departmental author
citation counts (not journal impact factors!) and RAE peer rankings
have been positive, sizable, and statistically significant for the
RAE, in all disciplines and all years tested. Variance there will be,
always, but a good-sized component from citations alone seems to be
well-established. Please see the studies of Professor Oppenheim and
others, for example as cited in:
Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online
RAE CVs Linked to University Eprint Archives: Improving the UK Research
Assessment Exercise whilst making it cheaper and easier. Ariadne 35.
http://www.ariadne.ac.uk/issue35/harnad/
> Human experts are necessarily selected from a population of experts, and it
> is often difficult to delineate between fields of expertise.
Correct. And the RAE rankings are done separately, discipline by
discipline; the validation of the metrics should be done that way too.
Perhaps there is sometimes a case for separate rankings even at
sub-disciplinary level. I expect the departments will be able to sort
that out. (And note that the RAE correlations do not constitute a
validation of metrics for evaluating individuals: I am confident that
that too will be possible, but it will require many more metrics and
much more validation.)
> Similarly, we
> know from quite some research that citation and publication practices are
> field-specific and that fields are not so easy to delineate. Results may be
> very sensitive to choices made, for example, in terms of citation windows.
As noted, some of the variance in peer judgments will depend on the
sample of peers chosen; that is unavoidable. That is also why "light
touch" peer re-validation, spot-checks, updates and optimizations on the
initialized metric weights are also a good idea, across the years.
As to the need to evaluate sub-disciplines independently: that question
exceeds the scope of metrics and metric validation.
> Thus, I am bit doubtful about your claims of an "empirically demonstrated
> fact."
Within the scope mentioned -- the RAE peer rankings, for disciplines
such as they have been partitioned for the past two decades -- there is
ample grounds for confidence in the empirical results to date.
(And please note that this has nothing to do with journal impact factors,
journal field classification, or journal rankings. It is about the RAE
and the ranking of university departments by peer panels, as correlated
with citation counts.)
Stevan Harnad
AMERICAN SCIENTIST OPEN ACCESS FORUM:
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/
UNIVERSITIES and RESEARCH FUNDERS:
If you have adopted or plan to adopt a policy of providing Open Access
to your own research article output, please describe your policy at:
http://www.eprints.org/signup/sign.php
http://openaccess.eprints.org/index.php?/archives/71-guid.html
http://openaccess.eprints.org/index.php?/archives/136-guid.html
OPEN-ACCESS-PROVISION POLICY:
BOAI-1 ("Green"): Publish your article in a suitable toll-access journal
http://romeo.eprints.org/
OR
BOAI-2 ("Gold"): Publish your article in an open-access journal if/when
a suitable one exists.
http://www.doaj.org/
AND
in BOTH cases self-archive a supplementary version of your article
in your own institutional repository.
http://www.eprints.org/self-faq/
http://archives.eprints.org/
http://openaccess.eprints.org/
Received on Mon Jun 16 2008 - 19:17:01 BST