On Tue, May 18, 2010 at 7:49 AM, Matthew Cockerill
<matt_at_biomedcentral.com> wrote:
> I'm guessing that you are making a ballpark estimate that the accuracy
> of the typical 'fact' in Wikipedia is around 50%, while the accuracy of
> a typical 'fact' in the peer-reviewed literature is around 99.95%.
That's right. But the figures are just guesses. The context was a
reply to Richard Poynder's pointing out that neither Wikipedia nor
peer reviewed journals were 100% reliable. I agreed, but noted that
that they differ greatly in terms of degree of unreliability.
> I don't think either of us has published a peer-reviewed systematic
> analysis on this, but my sense is that, if anything, the numbers could
> be the other way around.
I agree there are no objective data, but I greatly doubt the figures
are the reverse! The absolute and relative reliability estimates may
be wrong, but their direction, and the sizeable difference, are
unlikely to be wrong.
>The peer-reviewed original research literature is the bleeding edge of
> knowledge. As such, the original research literature inevitably contains
> myriad assertions that seem to be well supported at the time, but which
> turn out, on balance, to have been incorrect interpretations of the data
> or over-enthusiastic speculation.
Again, I don't know the absolute figures on this, but I greatly doubt
that WP entries on a topic will prove to be more reliable than
peer-reviewed entries on the topic. I would agree that peer-reviewed
*review articles* on a topic, written a few years later, are likely to
be more reliable than some of the original primary studies they
review, but I doubt that WP entries on the same topic will be as
reliable, on average, as either peer-reviewed primary studies or
peer-reviewed reviews (though individual WP entries do sometimes turn
out to be surprisingly reliable -- at least for a while, as long as
entry-rot does not set in).
> Wikipedia, via its strict 'no original research' policy, sets itself the
> more manageable (though still challenging) task of summarizing consensus
> opinion from a neutral PoV. As such, I think the facts/assertions it
> contains tend be much less disputed and subject to change than those in
> the original research literature.
They are indeed *supposed* to be written from a "neutral Point of
View." But they are written by anonymous authors who are answerable
only to other anonymous authors as well as anonymous adjudicators, all
of whom, equally strictly, are answerable only to one another, not to
expertise on the topic. The result is the unreliable (yet sometimes
surprisingly useful) mixture we get from WP.
I highly doubt that, on average, WP content is less "disputed" than
peer-reviewed content, whether primary or reviewed. And it is almost
axiomatic that it is far more "subject to change" (for better or for
worse).
(You seem to have a somewhat romanticized, if not ideologized PoV
about WP, Matt!)
> Obviously, it depends where in the vast resource you look.
That's what "reliability" means. For more reliable sources, it depends
less where you happen to be looking.
> But certainly, you must be using a particularly weird subset of Wikipedia if
> you are finding 50% of it to be factually 'wrong' in any significant
> way.
As I said, I am not referring to the hobbyist or trivial-pursuit
sectors of WP, but the scholarly/scientific entries. In the (few)
fields where I am myself a qualified expert, I find plenty of errors
and omissions. In the far larger number of entries on which I am not
an expert, I run into errors, about which I have the impression that
although they are not as rampant as in average journalism or most
blogs, they are far more frequent than in refereed research articles.
(And, I hope we agree, that's what the refereeing is done in order to
minimize.)
50% is probably an exaggeration, but the WP unreliability rate is
still, I think, substantially greater than the refereed research
unreliability rate -- though I agree that the right comparison is
probably with refereed secondary reviews rather than with individual
pieces of refereed primary research, since WP items are themselves
secondary (or tertiary) "reviews."
> Given Wikipedia aims to document disputed facts from a neutral POV, by
> describing the dispute, it is clear that if you hold a strong absolute
> opinions about which side of the debate is correct, you are unlikely to
> find Wikipedia expressing direct agreement with you on that topic. That
> is by design.
I am not talking about differences in opinion. And although there are
sometimes matters of opinion and bias in scholarly and scientific
research that refereeing sometimes fails to detect and correct, I
would have a lot more confidence in reports that have been adjudicated
by answerable experts (peer reviewers), themselves answerable to an
answerable meta-adjudicator (the editor) whose name, journal-title,
and track-record publicly under-writes its accepted publications than
to an anonymous sample of authors and adjudicators, answerable only to
one another's opinions.
Perhaps, Matt, you are unduly influenced here by the special case of
my "PoV" on the usefulness of calling the WP entry on OA "OA
(publishing)." I agree that my PoV on that is not neutral (which, by
the way, certainly does not mean it is wrong, since wrong/right is not
a matter of nose-counts -- or notability, or notoriety -- on such
questions).
But my matchbox estimates of reliability were not based on my own
experience in that special case of entry-(mis)naming. They are based
on my sample, across time, of WP entries in general, including those
on the (much rarer) cases where I am myself an expert on the topic in
question.
> Not only that, in the case of opinions which are disputed, you not only
> get to see what one specific opinionated expert finally decided, but via
> the discussion pages and the history, you get to see a summary of the
> different strains of thinking that have led to the current consensus.
Yes, yes. But for matters of fact in science and scholarship I'd
rather just have a reliable report of the facts, adjudicated and
reported by qualified and answerable experts, rather than the option
of delving through the sad history of anonymous, unanswerable, and
unqualified opinion, counter-opinion, and adjudication by those whose
adjudicative powers are grounded solely in having done a lot of
anonymous, unanswerable, and unqualified adjudication...
> Biology Direct's peer review model
> http://www.biology-direct.com/content/1/1/1
> and the open peer review on the medical BMC-series journals are both
> designed to provide a degree of the same kind of transparency regarding
> differences of opinion. In contrast, in most of the peer reviewed
> literature you see only the final end product, and unfortunately despite
> the best efforts of authors, peer reviewers and editors, it is
> inevitable that sometimes that final end product will turn out to be
> wrong.
The reliability, scaleability and sustainability of various "open peer
review" experiments are not yet known. Anonymity is a big difference
too.
(But I'd rather not re-open the much-aired question of peer review
experiments and reform in general here: It's already had a lot of air
time, and -- like publishing reform and copyright reform -- it tends
to co-opt time and attention from the practical goal of providing OA,
to which this Forum is dedicated. I realize I am the culprit who
introduced the somewhat off-topic question of the dynamics of WP, but
I was just hoping to elicit some reflections on direct experience from
a long-time AmSci Forum contributor [David Goodman] who went on to
become one of the adjudicators of WP. The AmSci Forum's focus is still
on freeing access to the peer-reviewed research literature, such as it
is, not to freeing it from peer review. The WP topic arose simply
because of WP's unfortunate contribution to the conflation of OA with
OA Publishing, by canonizing the conflation in its generic OA entry.
-- WP, by the way, generously gives OA two entries: you can look up
either "OA (publishing)" or "OA journal": from your PoV, Matt, that
may look like a good thing, but not from mine -- nor, I would argue,
OA's...)
Stevan Harnad
> -----Original Message-----
> From: American Scientist Open Access Forum
> [mailto:AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On
> Behalf Of Stevan Harnad
> Sent: 16 May 2010 16:11
> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
> Subject: Re: Wikipedia, Open Access and Cognitive Virology
>
> On Sun, May 16, 2010 at 9:12 AM, Richard Poynder
> <richard.poynder_at_btinternet.com> wrote:
>> SH: But Wikipedia's frequent accuracy and usefulness does not (in my
> view)
>> counterbalance the fact that it is frequently unreliable and
> inaccurate too
>> ...
>>
>> RP: Is that not also a reasonable description of the totality of the
>> research corpus today, and a feature of the peer-reviewed literature
> that
>> will become increasingly apparent as more and more of the content
> published
>> in the world's 24,000 scholarly journals becomes OA?
>
> It's a reasonable description of both corpora -- if you ignore the
> difference between (say) 50% unreliability and .05% unreliability...
>
Received on Wed May 19 2010 - 10:07:13 BST