On Wed, 20 Nov 2002, [identity removed] wrote:
> Dear Stevan,
>
> We are running a special report on the review of the RAE
> and you were one of the people suggested as having more lively,
> radical ideas about how research should be assessed in the future [rather
> than how the funding should be allocated]. I think HEFCE are a little
> disappointed by the replies they have had so far, which seem
> more about tweaking the current system than thinking 'out of the box'. We
> are asking a range of people what they would suggest. Would it be possible
> to talk to you about your ideas on this or perhaps you could email me
> a few words on how you think the current system can be changed?
>
> Best wishes [identity removed]
Happy to oblige.
To summarize, the UK is in a unique position -- for the very reason that
it is the only country with a national research assessment exercise like
the RAE -- to do two very closely related things in concert, with three
very likely and very positive outcomes:
(i) It will give the UK RAE a far more effective and sensitive measure
of research productivity and impact, at far less cost (both to the
RAE and to the universities preparing their RAE submissions).
(ii) Besides strengthening the assessment of UK research, it will
also greatly strengthen the uptake and impact of UK research, by
increasing its visibility, accessibility and usage.
(iii) At the same time, the UK RAE will thereby set an example to the
rest of the world that will surely be emulated, in both respects:
research assessment and research access.
The proposal is quite simple, though I will spell it out as a series
of 20 closely connected points:
(1) We already have an RAE, every 4 years.
(2) It costs a great deal of time and energy (time and energy that
could be used to actually do research, rather than preparing and
assessing RAE returns) to prepare and assess, for both universities
and assessors.
(3) It is no secret that for most areas of research, the single most
important and predictive measure of research impact is the so-called
"impact factor" -- the number of times a work has been cited (hence used)
by other research papers. This is a measure of the importance and uptake
of that research.
(4) The impact factor is used very indirectly in the RAE: Researchers each
submit 4 publications for the 4-year interval, and these are (informally)
weighted by the impact factor of the peer-reviewed journal in which
they appeared. (For books or other kinds of publications, see below;
in general, peer-reviewed journal- or conference-papers are the coin of
the research realm, especially in scientific disciplines.)
(5) If someone did a statistical correlation on the numerical outcome of
the RAE, using the weighted impact factors of the publications of each
department and institution, they would be able to predict the outcome
ratings quite closely. (No one has done this exact statistic, because
the data are implicit rather than explicit in the returns, but it could
be done, and it would be a good idea to do it, just to get a clear
indication of where the RAE stands right now, before the simple reforms
I am recommending.)
(6) There is no reason why the RAE should be based only on the impact
factors of 4 publications per researcher, nor why it should be weighted
by the impact factor of the journal in which it appeared, rather than by
the exact impact of each publication itself. (On average the two will
agree, but there is no reason to rely on blunt-instrument averages if
we can use a sharper, exact instrument: A researcher's individual paper
may have a much higher -- or lower -- impact than the average impact of
the journal in which it appears.)
(7) Nor is there any reason why the RAE should be done, with great
effort and expense, every 4 years!
(8) Since the main factor in the RAE outcome ratings is research impact,
there is no reason whatsoever why research impact should not be
continuously assessed -- and directly, rather than indirectly, via the
the true impact factor of the publication (or the author!), rather
than merely the journal's average impact factor.
(9) And there is now not only a method to (a) continuously assess full
UK research impact, and not only get this done (b) incomparably more
cheaply and less effortfully for all involved, while at the same time
making it (c) more sensitive and accurate in estimating the true impact
of the research, but doing the RAE this new way will also have a dramatic
effect on the magnitude of UK research impact itself, (d) increasing
research visibility, usage, citation and productivity dramatically,
simply by maximizing its accessibility.
(10) The method in question is to implement the RAE henceforth online
only, and the only two critical elements are (1) the submission of a
RAE-standardized online CV by every researcher and (2) a link in each
CV between every published paper -- books discussed separately below --
and the full digital text of that paper in that researcher's university
Eprint Archive (an online archive of that institution's peer-reviewed
research output).
http://www.eprints.org/self-faq/#institution-facilitate-filling
(See the free, open-source software we have developed at Southampton to
allow universities to create their own institutional Eprint Archives:
http://software.eprints.org/ )
(11) Currently, university peer-reviewed research output -- funded
by government research grants, the researcher's time paid for by the
researcher's institution -- is given, free, by all researchers, to the
peer-reviewed journals in which it appears.
(12) The peer-reviewed journals in turn perform the peer-review, which
assesses and improves the quality of the research (this is one of the
indirect reasons that the RAE depends on peer-reviewed journal
publications).
(13) There is a hierarchy of peer-reviewed journals, from those with the
highest quality standards (and hence usually the highest rejection rates
and impact factors) at the top, grading all the way down to the
lowest-quality journals at the bottom.
(14) The peer-reviewers referee for free; they are just the researchers
again, wearing other hats.
(15) But the it costs the journals something to implement the peer
reviewing. (Estimates are that it costs $200-$500 per paper.)
(16) Partly because of the cost of peer review, but mostly because of the
much larger cost of print-on-paper and its dissemination, journals charge
tolls (subscriptions. licenses, pay-per-view) for access to researchers'
research output (even though the researchers gave them the research for
free).
(17) The result of the access-tolls is a great loss of potential research
impact, because most institutions cannot afford the access tolls to most
peer-reviewed journals (there are 20,000 in all, across disciplines),
but only to a small and shrinking proportion of them.
(18) Hence the second dramatic effect of revising the RAE to make it
online continuous assessment based on the institutional self-archiving
of all UK peer-reviewed research output is that it will make all that UK
research accessible to all would-be users worldwide whose access is
currently blocked by access-toll-barriers.
http://www.publications.parliament.uk/pa/cm200304/cmselect/cmsctech/399/399we152.htm
(19) The UK full-text peer-reviewed research archives will not only be
continuously accessible to all potential users, but the access will be
continuously assessable, in the form not only of continuously updated
impact factors based on the classical measure of impact, which is
citations, but usage will also be measured at earlier stages than
citation, namely downloads ("hits"). And many other powerful new online
measures of research productivity and impact will develop around this UK
research corpus, increasing the sensitivity and predictiveness of the
RAE analyses more and more.
(See the online impact-measuring scientometric search engines we have
developed at Southampton:
http://citebase.eprints.org/cgi-bin/search
and
http://opcit.eprints.org )
(20) And all that is needed for this is for RAE to revert to online
submissions, requiring online CVs linked to the full-text draft of
each peer-reviewed publication in the researcher's institutional Eprint
Archive.
http://www.eprints.org/self-faq/#research-funders-do
Reference-link-based impact-assessment engines like citebase can then
be used by RAE to derive ever richer and more effective measures of
research productivity and impact, available to the RAE continuously. And
institutions could continuously monitor and improve their own research
productivity and impact, using those same measures. And the rest of the
world could see and emulate the system, and its measurable effects on
research visibility, uptake and impact.
Just a few loose ends: Books are often not give-aways, as peer-reviewed
research is, so this solution does not apply as well to the assessment
of book output -- but this was a problem even in the old RAE system,
because impact measures are not as readily available or widely used
for books. The new system will strengthen the RAE and its accuracy
and fairness in all sectors except books. And even with books there is
the option (especially with esoteric monographs that produce virtually
no royalty revenue) to put them in the Institutional Eprint Archives
too. And even if the book's full-text itself is not accessible online,
its metadata and references could be. Then the citation of books by
the online peer-reviewed publications will be a measurable and usable
estimate of their impact.
The UK is uniquely placed to move ahead with this and lead the world,
because the RAE is already in place. But we need to move fast,
because other countries are getting the idea too! The Netherlands has
no formal RAE yet, but it is about to implement a national system of open
research archiving for all of its universities called DARE:
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2356.html
It is just a matter of time before they realize that a marriage between
a national network of DARE-style institutional Eprint Archives and a
national RAE-style research assessment exercise makes a natural, indeed
an optimal combination. If/when they do, it will be they, and not the
UK, who lead the rest of the world toward this natural solution.
But although the naturalness and optimality -- indeed the inevitability
-- of all this is quite transparent, it is a fact that research culture
is slow to change of its own accord, even in what is in its own best
interests. That, however, is precisely why we have funding councils and
research assessment: To make sure that researchers do what is best for
themselves, and best for research, and hence also best for the supporters
(and beneficiaries) of research, namely, tax-paying society: The
institutional self-archiving of research output, for the sake of
maximizing research access and impact, has been much too slow in coming,
even though it has already been within reach for several years. The UK
and the RAE are now in a position to lead the world research community
to the optimal and the inevitable.
http://www.eprints.org/self-faq/#research-funders-do
We at Southampton, meanwhile, are continuing to try to do our bit to
hasten the optimal/inevitable for research and researchers. We are
planning to harvest all the metadata form the submissions to RAE 2001
http://www.hero.ac.uk/rae/submissions/ into RAEprints, a "meta-archive"
that is intended to demonstrate what RAE returns would look like
if this RAE upgrade proposal were adopted. Of course RAEprints (i)
will contain only four papers per researcher, rather than their full
peer-reviewed research output, (ii) it will only contain the metadata
for those papers (author, title, journal-name), not the full-text and
the all-important references cited. But we will also try to enhance the
demo by adding as much of this missing data as we can find on the Web,
so as to at least give a taste of the possibilities: Using paracite
http://paracite.eprints.org/ an on-line citation-seeker that goes out
and tries to find peer-reviewed full-text papers on the web, we will
"stock" RAEprints with as much as we can find -- and then we will invite
all the RAE 2001 researcher/authors to add their full-texts to
RAEprints too!
But we can't do it all alone. We hope HEFCE and RAE will put their full
weight behind progress toward this outcome, so beneficial to so many, in
so many ways.
Best wishes,
Stevan
-------------
Date: Fri, 25 Oct 2002 19:06:55 +0100 (BST)
From: Stevan Harnad <harnad_at_ecs.soton.ac.uk>
This is a response to the HEFCE "invitation to contribute"
recommendations for restructuring the RAE:
I have written two papers on how the RAE might be greatly improved in
its assessment accuracy and at the same time made far less effortful
and costly -- while (mirabile dictu) doing a great indirect service
to research and researchers, both in the UK and in the rest of the
scholarly/scientific world as well:
Harnad, S. (2001) "Research access, impact and assessment." Times Higher
Education Supplement 1487: p. 16.
http://cogprints.soton.ac.uk/documents/disk0/00/00/16/83/index.html
Harnad, S. (2001) The Self-Archiving Initiative. Nature 410: 1024-1025
http://www.nature.com/nature/debates/e-access/Articles/harnad.html
If you wish to see what the RAE would look like if UK research output
were continually accessible online, and hence continuously assessable, see:
http://citebase.eprints.org/
To see how the RAE could help hasten this outcome (which is in any case
optimal and inevitable), see:
http://www.eprints.org/self-faq/#research-funders-do
We at Southampton are currently harvesting the RAE submissions data
and putting them in an Eprint Archive to provide a "demo" of the sorts of
possibilities an online, open-access research corpus opens up for
research visibility, accessibility, uptake, usage, citation, impact and
assessability.
Best wishes,
Stevan Harnad
Received on Wed Nov 20 2002 - 15:36:57 GMT