On Wed, 5 Apr 2006, [identity deleted] wrote:
> Does that mean that if we found some metrics (physical
> characteristics, say) that correlated highly with a propensity to commit
> murder, we could safely lock these people up without the "wasteful
> motions" of a trial?
It is remarkable what strong reactions scientometrics elicit, very much the way
psychometrics do (IQ, and the G factor) in my own field.
There are several points one can make by way of reply:
(1) One can reject the criminal-trial analogy as far too shrill: Not only
are these not metrics on individuals but on groups, and not only are there
no comparably high metric correlates to predict crime, but if there were,
they would need to be prior performance-based to be comparable with the
case under discussion here. Even DNA metrics, though more favorable,
would be a false analogy; we are talking about performative predictors,
not constitutional ones. (Think of a long and exception-free history
of pedophilia, if you insist on a shrill analogy, and the question of
whether to release a seasoned repeat-offender into society on parole an
N+1st time.) But, basically, no light is shed by such far-fetched and
lurid comparisons: all they do is raise the temperature.
(2) RAE decisions are not made to the criminal-law criterion of "beyond
reasonable doubt" but to the civil-law criterion of "the vast preponderance
of the evidence." (It's not capital crime and punishment that's at issue here but
how best to disburse capital: public research funding.)
(3) Metrics of prior research performance are reasonable predictors of future
research performance; the rest depends on how much of the variance they account
for.
(4) Metrics do not become more reasonable when coupled with a
costly and time-consuming "jury trial" that consists of submitting 4
already-published articles -- already-peer-reviewed by expert juries
(their expertise co-varying somewhat with the reputation and rigour of
the courts in which the defendants were tried) -- to be re-tried in
a higher court consisting of small, generic panels that have neither
the expertise nor the time to re-do the original peer review properly,
and mostly just skim the already published papers, if that.
(5) If there is anything to the analogy of a lower court with a higher
court of appeal, then the tertium comparationis is that the higher court
does not re-do the original evidence-based jury trial, but weighs the
data (metrics, transcripts) therefrom on points of law. It would be as
absurd to start again from scratch at that level (except where a mistrial
has been declared) as it would be to re-review each published article
for the RAE (when it gives virtually the same outcome as the metrics
anyway).
If we are going to try metrics by analogy, may I suggest that we need
more rigour as to what analogies are admissible as evidence?
Stevan Harnad
Received on Wed Apr 05 2006 - 12:19:59 BST
This archive was generated by hypermail 2.3.0
: Fri Dec 10 2010 - 19:48:17 GMT