Here, reposted, is some feedback on meta-analysis from one of its
leading exponents:
Gene V Glass Says:
Mar 12, 2010 at 12:47 pm
http://scholarlykitchen.sspnet.org/2010/03/11/rewriting-the-history-on-access/
Far more issues about OA and meta analysis have been raised in this
thread for me to [be able to] comment on. But having dedicated 35
years of my efforts to meta analysis and 20 to OA, I can’t resist a
couple of quick observations.
Holding up one set of methods (be they RCT or whatever) as the gold
standard is inconsistent with decades of empirical work in meta
analysis that shows that “perfect studies” and “less than perfect
studies” seldom show important differences in results. If the question
at hand concerns experimental intervention, then random assignment to
groups may well be inferior as a matching technique to even an ex post
facto matching of groups. Randomization is not the royal road to
equivalence of groups; it’s the road to probability statements about
differences.
Claims about the superiority of certain methods are empirical claims.
They are not a priori dicta about what evidence can and can not be
looked at.
Glass, G.V.; McGaw, B.; & Smith, M.L. (1981). Meta-analysis in Social
Research. Beverly Hills, CA: SAGE.
Rudner, Lawrence, Gene V Glass, David L. Evartt, and Patrick J. Emery
(2000). A user's guide to the meta-analysis of research studies. ERIC
Clearinghouse on Assessment and Evaluation, University of Maryland,
College Park.
http://glass.ed.asu.edu/gene/resume2.html
On Thu, Mar 11, 2010 at 9:56 PM, Stevan Harnad <amsciforum_at_gmail.com> wrote:
> On Thu, Mar 11, 2010 at 12:17 PM, Philip Davis <pmd8_at_cornell.edu> wrote:
>
>> Stevan,
>> In my critique of this review today (see: http://j.mp/d91Jk2 ), I commented
>> on the inappropriate use of meta-analysis to the empirical OA citation
>> studies:
>>
>> "Meta-analysis is set of powerful statistical techniques for analyzing the
>> literature. Its main function is to increase the statistical power of
>> observation by combining separate empirical studies into one über-analysis.
>> It’s assumed, however, that the studies are comparable (for instance, the
>> same drug given to a random group of patients with multiple myeloma), but
>> conducted at different times in different locales.
>>
>> This is not the case with the empirical literature on open access and
>> citations. Most of the studies to date are observational (simply observing
>> the citation performance of two sets of articles), and most of these use no
>> statistical controls to adjust for confounding variables. Some of the
>> studies have focused on the effect of OA publishing, while others on OA
>> self-archiving. To date, there is still only one published randomized
>> controlled trial.
>>
>> Conducting a meta-analysis on this disparate collection of studies is like
>> taking a Veg-O-Matic to a seven-course dinner. Not only does it homogenize
>> the context (and limitations) of each study into a brown and unseemly mess,
>> but it assumes that homogenization of disparate studies somehow results in a
>> clearer picture of scientific truth."
>>
>> --Phil Davis
>
> Phil,
>
> Thanks for the helpful feedback.
>
> I'm afraid you're mistaken about meta-analysis. It can be a perfectly
> appropriate statistical technique for analyzing a large number of
> studies, with positive and negative outcomes, varying in
> methodological rigor, sample size and effect size. It is a way of
> estimating whether or not there is a significant underlying effect.
>
> I think you may be inadvertently mixing up the criteria for
> eligibility for meta-analysis with the criteria for a clinical drug
> trial (for which there rightly tends to be an insistence on randomized
> control trials in biomedical research).
>
> Now I would again like to take the opportunity of receiving this
> helpful feedback from you to remind you about some feedback I have
> given you repeatedly http://bit.ly/dkieVi on your own 2008 study --
> the randomized control trial that you suggest has been the only
> methodologically sound test of the OA Advantage so far:
>
> You forgot to do a self-selection control condition. That would be
> rather like doing a randomized control trial on a drug -- to show that
> the nonrandom control trials that have reported a positive benefit for
> that drug were really just self-selection artifacts -- but neglecting
> to include a replication of the self-selection artifact in your own
> sample, as a control.
>
> For, you see, if your own sample was too small and/or too brief (e.g.,
> you didn't administer the drug for as long an interval, or to as many
> patients, as the nonrandom studies reporting the positive effects had
> done), then your own null effect with a randomized trial would be just
> that: a null effect, not a demonstration that randomizing eliminates
> the nonrandomized drug effect. (This is the kind of methodological
> weakness, for example, that multiple studies can be weighted for, in a
> meta-analysis.)
>
> I am responding to your public feedback only here, on the SIGMETRICS
> list, rather than also on your SSP Blog, where you likewise publicly
> posted this same feedback (along with other, rather shriller remarks)
> http://j.mp/d91Jk2 because I am assuming that you will again decline
> to post my response on your blog, as you did the previous time that
> you publicly posted your feedback on my work both there
> http://bit.ly/8LK57u and here -- refusing my response on your blog on
> the grounds that it had already been publicly posted elsewhere
> (namely, here!)...
>
> -- Stevan Harnad
>
> PS The idea of doing a meta-analysis came from me, not from Dr. Swan.
>
>> Stevan Harnad wrote:
>>>
>>> ** Cross-Posted **
>>>
>>> [Note added by SH: These data are derived from Dr. Steve Hitchcock's
>>> bibliography of studies on the effect of open access and downloads
>>> ('hits') on citation impact. They are now ripe for a meta-analysis:
>>> You are encouraged to do one -- or to contact Dr. Swan and Dr.
>>> Hitchcock if you are interested in collaborating]
>>>
>>> ------------
>>>
>>> Swan, A. (2010) The Open Access citation advantage: Studies and
>>> results to date. Technical Report, School of Electronics & Computer
>>> Science, University of Southampton.
>>> http://eprints.ecs.soton.ac.uk/18516/
>>>
>>> ABSTRACT:
>>> This paper presents a summary of reported studies on the Open Access
>>> citation advantage. There is a brief introduction to the main issues
>>> involved in carrying out such studies, both methodological and
>>> interpretive. The study listing provides some details of the coverage,
>>> methodological approach and main conclusions of each study.
>>>
>> --
>> Philip M. Davis
>> PhD Student
>> Department of Communication
>> 301 Kennedy Hall
>> Cornell University, Ithaca, NY 14853
>> email: pmd8_at_cornell.edu
>> phone: 607 255-2124
>> https://confluence.cornell.edu/display/~pmd8/resume
>> http://scholarlykitchen.sspnet.org/author/pmd8/
>
Received on Fri Mar 12 2010 - 18:31:55 GMT