On Thu, Aug 28, 2008 at 10:15 PM, Phil Davis <pmd8 -- cornell.edu> wrote:
> Stevan,
>
> Because of the sheer number of articles published by PNAS,
> tracking the performance of each article was considered too
> onerous. As a result, I tracked the first and last 6-month cohort
> of articles (June-Dec 2004; and June-Dec 2006). By choosing the
> first and last cohort, I could estimate a temporal trend in the
> data. Please remember that PNAS was only one of the 11 journals
> analyzed in this study, and that Gunter Eysenbach's study (PLoS
> Biology, 2006) analyzed only a 6-month cohort in one journal
> (PNAS, June-Dec, 2006). Granted, a full dataset from PNAS would
> have been ideal, and I encourage you to gather and share the
> intervening years if you feel that the missing data points would
> change significantly the results of this study. My sense is that
> they won't, but will challenge you to prove me wrong.
Phil,
It's your call. Your preferred hypothesis about what causes the
Open Access Advantage is Self-Selection Bias. So if you want to
avoid Self-Selection Bias yourself, you have to design your
studies so as to try to falsify your hypothesis, rather than
trying only to confirm it.
Omitting the (self)-critical control conditions (the (1) non-randomized
control in the BMJ paper and the (2) citing-paper-date control
as well as the (3) self-archived-article control in the
JASIST paper), omitting the further data-analysis (of (4) the
PNAS data-set and the (5) journal and article citation levels
in the JASIST paper), and omitting the further data-gathering
time-base (beyond the (6) first year in the BMJ study) is not the
way to make a credible case for your hypothesis (for those
who did not already believe that the OA Advantage was just a
Self-Selection Bias!).
If I had been the referee for BMJ or JASIST (having first fully
disclosed that my own preferred hypothesis about what causes
the Open Access Advantage is in conflict with yours), these are
the six further tests I would have recommended, in order to avoid
selective confirmation bias.
(It would not have been an altogether suitable reply to my referee
report, I think, to suggest that if I thought they would make a
difference, then I should be the one to go and do them!)
Chrs, Stevan
"Davis et al's 1-year Study of Self-Selection Bias: No Self-Archiving
Control, No OA Effect, No Conclusion"
http://openaccess.eprints.org/index.php?/archives/441-guid.html
"Confirmation Bias and the Open Access Advantage: Some Methodological
Suggestions for Davis's Citation Study"
http://openaccess.eprints.org/index.php?/archives/451-guid.html
> Davis, P. M. (2008 in press). Author-choice open access
> publishing in the biological and medical literature: a citation
> analysis. Journal of the American Society for Information Science
> and Technology. http://arxiv.org/abs/0808.2428
>
>
> Stevan Harnad wrote:
>
>> How does the fact that the overall sample was small and the
>> PNAS sample was large justify that the entire PNAS data-set was
>> not analyzed? (I don't contest that it should be analyzed (i)
>> within the aggregate as well as (ii) separately, and that (iii)
>> the rest should also be analyzed separately too, to avoid
>> skewing, I just don't understand why the full analyses were not
>> done and their results reported.)
Received on Fri Aug 29 2008 - 12:48:13 BST