I don't disagree with Stevan here, nor with his earlier comments that we
need to concentrate on filling up the "empty space" of repositories. The
crucial issue here is the incentive system to do so. Unless we change
the reward mechanisms or incentives, there will not be major academic
movements in the short term.
I copy below the first section of a September first draft for the OACI
Working Group which met in Leiden. To ignore ISI and indeed Elsevier's
Scopus would be unwise. What we need to do is to identify their metric
"deficiencies" in terms of bibliometric cleansing, coverage, etc; to
lobby those bodies which are adopting ISI and other rankings
unreservedly; to promote the wider bibliometric web measurements listed
by Stevan and Tim Brody; and most importantly, get them incorporated
into the metrics of the research evaluation processes which are driving
the academic community. See The Guardian cartoon, October 23, attached
which, while not complex, puts the point
http://books.guardian.co.uk/posysimmonds/page/0,12694,1334031,00.html
Herewith the beginning of my original OACI draft:
"How is research excellence measured? Tijssen has analysed the issues
and challenges in utilising "measurable attributes" to establish
"scoreboards of research excellence". (Tijssen, 2004) This is an issue
which has increasingly interested governments, universities and funding
bodies as measures of accountability and quality are sought.
Professor Sir David King, the Chief Scientific Advisor and Head of the
UK Office of Science and Technology has stated that the "ability to
judge a nations scientific standing is vital for the governments,
businesses and trusts that must decide scientific priorities and
funding. (King, 2004). Using only ISI data, which King says "provides
metrics for judging achievement", he recently evaluated the UK's
performance in science and engineering
Professor Tony Hey, the Chair of the UK eScience Program has stated in
an email to Professor Stevan Harnad (circulated to the OACI group 9
September)" The Government is keen to measure the effectiveness of the
UK's research funding by comparing the citation impact of UK researchers
field by field. These citation counts come, of course, from ISI and the
US is top in almost all fields. The UK is second in some but only 3rd or
4th in others like engineering. I suspect that the ISI ratings may have
a bias towards US researchers and US journals and wondered if OA
citation measurements give the same or similar rankings. Obviously at
this stage of institutional repositories this is very early days for
such analyses"
Borgman and Furner in a comprehensive review article "Scholarly
Communication and Bibliometrics" provide references showing how
bibliometric methods are used to assess the research performance of
entire nations. (Borgman and Furner, 2002). Surprisingly, the United
States nationally seems not to have engaged in such direct national
dialogues, perhaps because it has no direct national equivalent of the
RAE's for universities or perhaps because American universities usually
figure quite prominently in global rankings and therefore no problem is
perceived. However, King, in his Nature article, notes that US
scientific output has declined proportionally in comparison to that of
the UK. Dr Diana Hicks has also indicated an absolute decline in the
number of US papers in international peer reviewed literature. (Hicks,
2004).
Sir Gareth Roberts, Chair of the UK 2008 RAE Exercise, in a June 2004
presentation to the Australian Scholarly Communication Round Table,
indicated that the purpose of RAE's was to allow funding bodies to
assess the quality of research arising from the investment of public
money; enabling the academic sector to assess its success; and inform
its future strategy and perhaps most importantly, to inform a funding
model. (Roberts, 2004).
The Berlin 2 Roadmap wishes to "discredit impact factor as appropriate
measure in career evaluation and tenure promotion",
(
http://www.zim.mpg.de/openaccess-cern/presentation-oa2berlin-roadmap-pro)
but seems to ignore the political reality?
Internationally research assessment and quality exercises have come to
rely increasingly on metrics such as the ISI citation indexes. The ISI
Citation data is invaluable to such bodies, as they allegedly "provide a
quick and easy yard stick for measuring research quality". (Adam, 2002)
The Shanghai Jiao Tong Index and University League Tables
http://ed.sjtu.edu.cn/ranking.htm
The Shanghai Jiao Tong Index of the "Academic Ranking of World
Universities" is perhaps the prime example of the importance assumed by
universities in a competitive environment. Thus the latest release of
the Shanghai rankings was picked up, for example, in August 2004 in New
Zealand, UK, Australian Canadian, Taiwanese and European, newspapers in
terms of "boasting" respective individual and thus national research
rankings.
Professor Anthony F J van Raan of Leiden University has shown in a most
cogent paper, "Fatal Attraction" that the bibliometric underpinnings of
the Shanghai rankings are far from definitive. (Van Raan, 2004). While
there are significant issues to be addressed in the collection and use
of bibliometrics there is just as important an issue to be addressed,
namely the simplistic acceptance by governments of "excellence" in terms
of usage of citation analyses.
The Australian National Academies Forum in July 2004:
http://www.science.org.au/conferences/researchexcellence/index.htm,
which was attended by the nations top scientists, research council
heads, government administrators, etc highlighted that the vast majority
of the OACI issues "we" are talking about are mostly unfamiliar to such
groups. They are happy to accept what they believe are definitive
indicators of research excellence, unaware of the concerns expressed in
bibliometric and scholarly communication circles. The main issue in this
context is not the use of bibliometric indicators per se, but the
application of inadequate or flawed bibliometric measures. Again rather
than deride this approach," "we" need to be aware of the political
realities and respond accordingly.
Van Raan says rankings such as the Shanghai one "are part of a larger
problem in the science evaluation circus. Quite often I am confronted
with the situation that responsible science administrators in national
governments and in institutions request the application of bibliometric
indicators that are not advanced enough. They ...want to have it 'fast',
in 'main lines', and not 'too expensive ... the fault of these leading
scientists and administrators is asking too much and offering too
little." (Van Raan, 2004).
Van Raan continues that "it is not so much the commercialization of the
monopolist data producer ISI that makes the problems. These heads of
institutions, government administrators and policy makers are the first
to blame that the intermediary research groups that hitherto cleaned the
crude ISI data, prepared the data for the construction of reliable
bibliometric indicators and developed the competence and skill to
interpret the indicators (Weingart 2004) are being squeezed out of the
market".
Since the purchase of ISI by Thomson Corporation in the early 1990s
there has been a marked change in the marketing and commercialisation of
the product. One commentator has called the pre-Thomson purchase period,
ISI's "Romantic Period", when specialists in the field were allowed
greater freedom with data manipulation. (Braun quoted in Adam, 2002).
Since the purchase by Thomson there has undoubtedly been a significant
increase in the influence of ISI, due partly through a combination of
increased market dominance, as well as acceptance, as evidenced above,
by administrators of what to them is a ready made metric as a tool for
performance analysis.
Dr Paul Wouters has stated in the Abstract to his seminal thesis, The
Citation Culture that the "Science Citation Index is moreover not merely
a bibliographic instrument. It also creates a new picture of science via
bibliographic references found in scientific literature. In this way,
the SCI provides a fundamentally new representation of science. By
focussing on the seemingly most insignificant entity in scientific
communication, the inventors of the SCI have created a completely novel
set of signs and of a new symbolic universe." (Wouters, 1999).
It was quite clear from Sir Gareth Roberts's presentation in Australia
to the NSCF Roundtable that for the UK RAE, that in certain science
disciplines the existing citation data, ie the ISI data, could be taken
at face value. This reaffirms the power of the ISI citation indexes
because to the administrators it is a ready made tool for measurement,
although they are often unaware of the problems in the use of the data,
eg the need for bibliographical cleansing, the differences between
different disciplines, the lack of coverage of certain subjects, author
self-citation patterns, etc."
The OACI Working Group is pursuing this topic and hopefully will issue a
"Leiden Declaration" incorporating an Open Access Impact Factor.
Colin Steele
Received on Sun Oct 24 2004 - 21:18:24 BST