Re: "Academics strike back at spurious rankings" (Nature, 31 May)

From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
Date: Sun, 3 Jun 2007 16:07:13 +0100

On Sun, 3 Jun 2007, Loet Leydesdorff wrote:

> OK: The validation is the measure.

No, the validation is something that one does in order to establish the
reliability and predictive power of one's measures (metrics).

One takes the measure (or measures) one seeks to validate and first tests
their internal reliability (by autocorrelation) and then their external
validity, by testing their correlation with an external criterion
or face-valid measure of what one is trying to measure or predict.

If one wished to validate barometric pressure as a predictor of rain,
one would first check its reliability (does it give the same value if
measured repeatedly?) and then, if is reliable, one checks its validity
(how closely does it correlate with subsequent rainfall)? Once its
validity is established, one can use pressure to predict subsequent
rain.

In the case of OA metrics, the idea is not to validate merely one
predictor metric, but a weighted battery of diverse metrics, for greater
joint predictive power. These not only have to be validated against
face-valid human criteria or other validated metrics, but they have to
be validated field by field, and application by application (depending
on what criteria one is trying to predict and evaluate).

And multiple regression determines what percentage of the variance in
the criterion each predictor metric accounts for.

> Thus, we would take a number of predictive variables x1, x2, etc. (e.g.,
> journal impact factors, total citations, number of publications, nation (!),
> etc.) and then fit the outcome to the expert opinions (y) so that:
>
> y = a * x1 + b * x2 + c * x3 + .....
>
> Is that the idea of the sophisticated (OA) scientometrics?

Not quite. "Nation" is not a predictor variable, any more than
individual or institution is: they are just potential contexts
for comparison, if we were interested in comparing nations (or
institutions, or individuals).

And OA scientometrics are not sophisticated in the sense of depending on
new, sophisticated statistical techniques -- multiple regression, after
all, is quite classical -- but in being based on a far larger and richer
set of metrics than classical citation analysis, thanks to the OA
database.

Stevan Harnad
Received on Sun Jun 03 2007 - 18:31:37 BST