On Mon, 20 Mar 2006, C.Oppenheim wrote:
> RAE panels DO consider anything and everything; it's not just rhetoric.
> It's not true that RAE panelists rely on the track record of the journal
> that the material appeared in, and I'm glad that this is indeed the way the
> RAE works because journal referees can and do sometimes make mistakes when
> they accept an article for publication.
We are now disagreeing both on points of fact and on points of
practicality and logic.
No one, of course, other than the individual evaluators themselves, can
know what they actually do do: How many papers they actually read, how
carefully and competently, and whether or not they are influenced,
consciously or unconsciously, by the name and track record of the
journal in which each article appeared. (I am quite ready to believe
they do *not* count citations!)
Let us not spend more time on disagreeing about these private
psychological facts about which no other party can have privileged
knowledge (except in his own individual case, if he served as an
RAE panelist). On paper, anything and everything is welcome for
RAE submissions and everything and everything is evaluated. That
formal stipulation is uncontested.
Now to the practicality and the logic, on which I invite everyone to
reflect and make their own judgement:
(1) About 175 UK institutions submitted 4 items each, for each unit
of assessment (about 70) in 2001
http://www.hero.ac.uk/rae/
(2) That means about 700 papers need to be re-evaluated by the
panels in each unit of assessment (total: 49,000 UK papers to be
re-evaluated)
(3) The peer-reviewed papers among them (the vast majority, I would
suggest) have already been evaluated by qualified experts -- their
degree of expertise and qualification being related to the established
and known track record of the journal in which they were published,
the better journals being the international ones, drawing on worldwide
peer expertise in the specific topic of the paper.
Now the (a) practical and (b) logical questions we should put to
ourselves in trying to second-guess what the individual evaluators can
and do actually do:
(a) Do the 70 RAE panels have the time and peer expertise for each of
their 700 papers to re-do the evaluation that the peer reviewers
have already done?
(b) Is it logical even to imagine that they would re-do it all
while ignoring the fact that it has *already been done,* by the
peer reviewers?
(The idea that this whole peer exercise would be re-done on the off-chance
that it might remedy a peer error in the original peer review is just --
at least to my ears -- so unrealistic and far-fetched as to border on
the absurd. It does seem, however, to have a ring of resonance with a
bit of local folk mythology about a kindred bit of ergonomic profligacy
much-entrenched in these parts: 2nd marking of essays and examination
results!)
> Panels will of course have
> different policies re what proportion of material they will read in detail,
> but I would be surprised if a major criterion was "this appeared in Nature
> so we don't need to read it".
No doubt. But how, dear Charles, do you imagine that the papers -- for
which we studiously ignore the track record of the journal in which
they appeared and dutifully refrain from counting their citations *and*
we don't read them "in detail" -- are to be evaluated and ranked?
> My research has shown a **strong correlation** between citation counts **of
> individuals in a department** and RAE score for department that they work
> in, but has shown **no correlation whatsoever** between **impact factor of
> journals** submitted and RAE score obtained by department. Departmental
> heads should NOT rely on journals with high impact factors to get a good RAE
> score.
The logic of this entirely escapes me, since (1) total citation scores
for individual articles are correlated with the average citation scores
of the journals they were published in (though I agree completely that
individual citation counts are the sharper instruments and journal
averages are the blunter instrument) and (2) how do you imagine that
such a powerful correlation would percolate through to the RAE rankings
if the journal name and citation impact are ignored and a "proportion"
of the papers is not "read in detail"?
(But I would like to ask an empirical question of detail: Are you saying
that your studies compared (a) the correlation between individuals'
articles' citation counts [I] and their departmental RAE ranks [R] and
(b) the correlation between individuals' articles' journal citation
(average) impact factors [J] and their departmental RAE ranks [R]
and found that IR correlation was high and JR correlation was
zero? I would have expected the correlation to be positive in both cases,
but higher for IR. The celebrated 2002 Smith/Eysenck study looked only
at IR, not JR:
http://cogprints.org/2749/)
> So whilst I agree with Stevan that preferably refereed material should be
> submitted, that doesn't mean one should refuse to submit non-refereed
> material or material from a low impact factor journal.
I don't think departments refuse to submit articles from low-impact-factor
journals to RAE (although they wisely prefer to submit articles from
high-impact-factor journals whenever they can, and those are indeed more
likely to lead to a higher RAE rank). But the issue we were discussing
was (i) whether departments refuse to submit *unpublished* papers to RAE
(and I hold by the hunch that where they can refuse, they do) and (ii)
whether an unpublished paper deposited in an IR counts as a published
paper (it doesn't, either for the RAE or any other academic purpose).
Chrs, Stevan Harnad
> ----- Original Message -----
> From: "Stevan Harnad" <harnad_at_ECS.SOTON.AC.UK>
> To: <JISC-REPOSITORIES_at_JISCMAIL.AC.UK>
> Sent: Monday, March 20, 2006 12:19 PM
> Subject: Re: RAE and IRs
>
>
> > Dear Rachel, I am afraid we are going around in circles:
> >
> > It is uncontested that RAE formally states:
> >
> > (1) You can submit anything
> > (2) We will evaluate everything
> > (3) We don't count citations
> >
> > But the fact is that (1) is untrue, because your department will not
> > (sensibly, mercifully) let you submit just anything. It won't even
> > submit you as an active researcher if you can't come up with 4 items that
> > meet their criteria (which are, in many fields, sensibly: articles in
> > high-impact refereed journals and conference proceedings).
> >
> > And the fact is that (2) is not only untrue, but absurd, because RAE
> > has neither the time nor the expertise to (re)-evaluate everything;
> > hence the only sensible thing to do (and they do it) is for the RAE
> > evaluators to rely on the track-record of the journal that peer-reviewed
> > the submitted article.
> >
> > And the fact is that although it is true that RAE do not count citations
> > explicitly (3), both the (sensible and realistic) departmental stress
> > on high-impact peer-reviewed journal submissions (1) and the necessary
> > (and sensible) taking-into-account by the RAE evaluators of the track
> > record of the peer-reviewed journal in which an article appeared (2)
> > (rather than any absurd and quixotic attempt to re-do peer review in-house
> > for all of the UK's research output!) result in the many times reported
> > outcome that the RAE rankings are highly correlated with (hence highly
> > predictable from) citation rankings.
> >
> > http://www.ariadne.ac.uk/issue35/harnad/
> >
> > I have several times promised to bow out of this discussion, but I do find
> > it mind-boggling how it manages to keep getting bogged down in matters of
> > superficial form while completely missing matters of substantive content!
> >
> > I've gone on to re-annotate the water that has already gone under the
> > bridge in this meandering discussion (below) to try to spare some of
> > us from just repeating history:
> >
> > On Mon, 20 Mar 2006, Rachel Bruce wrote:
> >
> >> Going back to the discussion on RAE rules re publication I have spoken to
> >> a
> >> colleague at HEFCE to get their view:
> >>
> >> RAE Guidance on submissions (RAE 03.2005) allows any form of publicly
> >> available assessable output to be submitted as well as confidential
> >> reports
> >> which may be submitted if the HEI has prior permission that the outputs
> >> may
> >> be made available for assessment.
> >
> > May be made available. But will they get you a good rank? And will your
> > department allow you to submit just anything (at the risk of its rank)?
> >
> >> A proportion of outputs submitted to the
> >> RAE are not text-based, and hence the RAE refers to outputs rather than
> >> publications.
> >
> > Irrelevant to the matter under discussion.
> >
> >> In the generic statement on criteria and working methods (at the
> >> beginning
> >> of each set of subject based panel criteria), it is clearly stated that
> >> "all
> >> forms of research output will be assessed on a fair and equal basis.
> >
> > A pious platitude, with no substance, one way or the other.
> >
> >> Sub-panels will neither rank outputs, nor regard any particular form of
> >> output as of greater of lesser quality than another per se.
> >
> > Utter, abject nonsense, immediately contradicted by the next sentence.
> >
> >> Some panels may
> >> specify in their criteria that where they do not examine an output in
> >> detail
> >> , they may use, as one measure of quality, evidence that the output has
> >> already been reviewed or refereed by experts (who may include users of
> >> research) and has been judged to embody research of high quality.
> >
> > An otiose and long-winded way of saying "please ignore what I just
> > said in the previous sentence."
> >
> >> no panel
> >> will use journal impact factors as a proxy measure for assessing
> >> quality."
> >> (paragraph 32 of the Generic statement (RAE 01.2006)).
> >
> > "And just in case that has not confused you sufficiently, I shall now
> > contradict it again."
> >
> > (In taking the journal's name and quality track-record into account,
> > you are implicitly influenced by its impact factor, whether or not you
> > do an explicit calculation -- which the RAE does not do. [Btw, if it had
> > any sense, the RAE *would* count submitted research/researcher citations
> > explicitly -- not journal impact factors, which are blunt instruments,
> > being merely the average for the journal: That way RAE would realise how
> > much it is already relying on citation impact implicitly, and it might
> > at also scrap a lot of useless make-work and time-wasting on the part of
> > both the submitters and the evaluators!)
> >
> >> From: Morag Greig, 16 March 2006 12:43
> >>
> >> At Glasgow we found that particularly in the initial stages of setting up
> >> our repository some academics were under the impression that we were
> >> asking
> >> them to stop publishing in peer reviewed journals and to simply make
> >> their
> >> research available in the repository.
> >
> > Solution: Correct the impression: They are being asked to make their
> > published, peer-reviewed research available in the repository.
> >
> >> This lead to concerns that such a move
> >> would have an adverse impact on performance in the RAE, as there seemed
> >> to
> >> be a fairly widespread belief that the RAE panels would consider such
> >> 'publications' of low value.
> >
> > The widespread belief is correct, but since no one is proposing
> > to submit unpublished work to RAE, but simply to deposit published work
> > in the IR (for submission to the RAE), nothing is at issue.
> >
> >> The issue of making publications being submitted to the RAE available in
> >> a
> >> repository is different, as here I assume you would be talking about
> >> material that had been published in a peer-reviewed journal, conference
> >> proceeding etc..
> >
> > Both published and unpublished work can be deposited in an IR, but
> > published, peer-reviewed journal articles are the primary, high-priority
> > target.
> >
> > Submitting to the RAE via IRs is a separate matter, but there too,
> > published, peer-reviewed journal articles are the primary target.
> >
> >> From: John Murtagh, 16 March 2006 10:32
> >>
> >> Interesting debate on the Research Assessment Exercise and IRs.
> >> From the responses so far I tend to agree with John Smith at Kent that
> >> unpublished material is (or will in the future) be as much worth to
> >> research output for an institution as anything else, as long as it is
> >> used and cited. That this shouldn't necessarily be a so-called vanity
> >> project.
> >
> > This is an unrealistic fallacy. Vide supra.
> >
> >> "...once we move away from the concept of 'impact factors' and use
> >> individual article citation analysis instead,
> >
> > So far, so good, since individual citation counts are a sharper
> > instrument than journal averages (though the journal deserves a
> > separate component in the regression equation too).
> >
> >> peer review will merely
> >> be the first stage (and not necessarily the most important) in assessing
> >> the real value of someone's work."
> >
> > Nonsense in the case of most work -- though no doubt some values (or
> > errors) will come out in the post-peer-review wash too, in some cases,
> > individual peer judgement, be it ever so expert, being fallible too...
> >
> >> I was interested to know from Morag Greig's response about how the RAE
> >> factored into the "confusion" of academics who were unwilling to
> >> deposit into the IR because of it not being a substitute for
> >> peer-review.
> >
> > Please, let us now dispel this confusion, rather than compounding it.
> >
> >> Could Morag describe this a little more? I would like to use the RAE
> >> exercise as an inducement, but it sounds like a deterrent from
> >> depositing (in the gathering of materials at least).
> >
> > RAE is an inducement to depositing in an IR: It's not only a more
> > efficient way to submit to RAE, but self-archiving (early!) raises
> > research impact, hence RAE rating.
> >
> > http://opcit.eprints.org/oacitation-biblio.html
> >
> >> From: Morag Greig, Wed, 15 Mar 2006
> >> Subject: Re: Generic Rationale and Model for University Open Access
> >> Mandate
> >>
> >> It's worth pointing out that many repositories have gone to some
> >> lengths to explain to authors that repositories are (in most cases) NOT a
> >> substitute for publication in a peer reviewed journal (whether that be an
> >> OA
> >> journal or a subscription journal). Confusion over this issue has lead to
> >> many
> >> academics being unwilling to deposit in their institutional repositories.
> >
> > Hear, hear!
> >
> >> From: Leslie Carr
> >>
> >> Whether articles are disseminated by an institutional repository is
> >> irrelevant to RAE assessment.
> >>
> >> It is whether they have been published by a journal (or conference or
> >> whatever the assessment panel uses as a criterion) that matters.
> >>
> >> In other words, appearing in an IR does not make an article
> >> "published" in the eyes of the RAE.
> >
> > Spot-on (for those with eyes to see and ears to hear!).
> >
> >> On 15 Mar 2006, at 09:50, Wolfgang Greller wrote:
> >>
> >> Can anyone tell me whether articles published in an institutional e-
> >> print repository count as publication in UK RAE terms?
> >
> > No it doesn't.
> >
> >> From: "C.Oppenheim"
> >>
> >> I don't agree with Les.
> >>
> >> Panels are instructed to consider *any* sort of published output, and
> >> this would consider something that had only ever appeared on an IR, or
> >> on a web page; indeed, statistics from the last RAE show that a small
> >> proportion of the items examined by the panels were in web page form
> >> only. However, if you want to impress the RAE panel, the item should
> >> have been through a proper refereeing process.
> >
> > Charles later conceded that he was speaking about RAE's formal criteria,
> > not actual submission practice, let alone successful/optimal practice
> > (in terms of RAE outcome), by departments.
> >
> >> From: Stevan Harnad
> >>
> >> All this means is that RAE also considers unpublished papers. It has
> >> nothing whatsoever to do with whether or not those papers were self-
> >> archived on the web. (Plus, as we know, and as many studies by
> >> Charles himself has shown, the RAE outcome is highly correlated with
> >> the citation counts of the journal articles submitted; and
> >> Departments place high -- too high -- emphasis on the "citation
> >> impact factor" of the journal in which the 4 articles selected for
> >> submission were published.)
> >>
> >> So although the RAE nominally considers all papers, and it definitely
> >>
> >> does not count citations directly, there is considerable emphasis on
> >> journal articles on both sides. Moreover, I profoundly doubt that
> >> many RAE submissions actually get read (let alone read by someone as
> >> competent to evaluate them as the referees of the journal in which
> >> the originally appeared). So the fate of an unpublished manuscript
> >> amounts to the fate of a book judged by its cover (without even the
> >> help of its publisher and his prior track record for quality)...
> >
> > Amen
> >
> > Stevan Harnad
> >
>
Received on Mon Mar 20 2006 - 15:03:28 GMT