Peer Review Selectivity Determines Quality, Not Open Access vs. Toll Access

From: Stevan Harnad <amsciforum_at_GMAIL.COM>
Date: Fri, 21 Nov 2008 22:55:41 -0500

    [ The following text is in the "WINDOWS-1252" character set. ]
    [ Your display is set for the "iso-8859-1" character set. ]
    [ Some characters may be displayed incorrectly. ]

In "Open Access: The question of quality," Richard Poynder writes:
      "Open Access scientometrics... raise the intriguing
      possibility that if research becomes widely available on
      the Web the quality of papers published in OA journals
      may start to overtake, not lag, the quality of papers
      published in TA journals... Why? Because if these tools
      were widely adopted the most important factor would no
      longer be which journal you managed to get your paper
      published in, but how other researchers assessed the
      value of your work ? measured by a wide range of
      different indicators, including for instance when and how
      they downloaded it, how they cited it, and the different
      ways in which they used it."

All true, but how does it follow from this that OA journals will
overtake TA journals? As Richard himself states, publishing in an OA
journal ("Gold OA") is not the only way to make one's article OA: One
can publish in a TA journal and self-archive ("Green OA"). OA
scientometrics apply to all OA articles, Green and Gold; so does the
OA citation advantage. 

Is Richard perhaps conflating TA journals in general with top-TA
journals (which may indeed lose some of their metric edge because OA
scientometrics is, as Richard notes, calculated at the article rather
than the journal level)? The only overtaking I see here is OA
overtaking TA, not OA journals overtaking TA journals. (Besides,
there are top-OA journals too, as Richard notes, and bottom-rung TA
ones too.)

It should also be pointed out that the top journals differ from the
rest of the journals not just in their impact factor (which, as
Richard points out, is a blunt instrument, being based on the journal
average rather than individual-article citation count) but in their
degree of selectivity (peer revew standards). If I am selecting
members for a basketball team, and I only accept the tallest 5%, I am
likely to have a taller team than the team that is less selective on
height. Selectivity is correlated with impact factor, but it is also
correlated with quality itself. The Seglen effect (that about 80% of
citations go to the top 20% of articles) is not just a within-journal
effect: it is true across all articles across all journals. There is
no doubt variation within the top journals, but not only are their
articles cited more on average, but they are also better quality on
average (because of their greater selectivity). And the
within-journal variation around the mean is likely to be tighter in
those more selective journals than the less-selective journals.

OA will give richer and more diverse metrics; it will help the cream
(quality) to rise to the top (citations) unconstrained by whether the
journal happens to be TA or OA. But it is still the rigor and
selectivity of peer review that does the quality triage in the
quality hierarchy among the c. 25,000 peer reviewed journals, not OA.

(And performance evaluation committees are probably right to place
higher weight on more selective journals -- and on journals with
known, longstanding track-records.)

Stevan Harnad
Received on Sat Nov 22 2008 - 04:00:16 GMT