On Mon, 10 Apr 2000, sterling stoudenmire wrote:
> Prof. Stevan Harnad I am a CPA! The Green article which you quoted below,
> raises several questions as to standards of peer review.. What are the
> rules of peer review? Are they uniformily applied between journals? How is
> a reader of the peer reviewed journal or article to know that the article
> was peer reviewed in accordance with adequate standards?
>
> I wonder should an association be created for developing standards of peer
> reviews. Then attesting, publisher conformance to the standards by an
> independent audit. The audit opinion might state something like the
> following:
>
> you may forward this on if you like it. [text appended below]
I'm afraid peer review is a bit more complicated than what you propose
below. The meaning of "peer" is "qualified specialist." The only ones
in a position to "audit" the specialists are further specialists; there
is no generic perspective here.
So before you could set up an evaluation system for peer evaluation,
you would have to set up an evaluation system for the evaluation of
peer evaluation -- in every specialty. All worthy ideas, but all
require prior experimental testing, not a priori implementation.
And, frankly, peer evaluators are a rare resource (us), with many other
demands on our time; and refereeing is unpaid (and there isn't faintly
enough money to pay us so much that we'd rather do THAT than our
primary calling: research; nor would I expect that those who would
readily give up their research time if paid to do evaluator-evaluation
would be the experts I'd trust most... but these too are empirical
rather than a priori matters).
Stevan Harnad
On Mon, 10 Apr 2000, sterling stoudenmire wrote:
> Addressed to the
> Board of Directors of
> The Publisher of XYZ Journal
> 69 Hideway Lane
> Bogota, Somewhere Down South, -000110--0000-90
>
> I (WE) have audited the peer reviews made by the XYZ journal for the period
> beginning (begindate) and ending (enddate) as listed in the Articles
> Published Report for the same period in accordance with the standards
> established by the International Peer Review Audit Standards
> Institute(IPRASI).
>
> In my opinion, as an auditor, in good standing with both the American
> Institute of Certified Public Accountants and the International Peer Review
> Audit Standards Institute (IPRASI), I find the XYZ journal has selected
> content, established prior knowledge by adequate reference, maintained its
> presentation standards, and performed quality reviews by qualified subject
> matter peers in a consistent and uniform manner in compliance with the
> standards established by IPRASI, except as noted in paragraph 2 and except
> as included in the Publishers Notes to the Articles Published Report.
>
> Paragraph 2 might say something like this
>
> Article 12, in the Articles Published Report, fails to disclose that one
> author is a brother in law of the publisher of the XYZ journal hereto and
> the brother in law is the president of a company that accounted for 40% of
> the total revenue of the journal during the publication period, and during
> each of the two past publication periods, the same company accounted for
> 65% and 90% respectively of the revenues to the XYZ Journal. Furthermore,
> the publisher is highly dependant upon revenues from the above cited
> company for its overall operations.
>
> Signed, dated, Auditor's Name, CPA, IPRASI
>
> tne next page should be the Articles Published Report.
>
> Articles Published Report
> XYZ Publisher
> Period Beginning ( Startdate) to Ending (Enddate)
>
> It should be addressed to the general audience of the Jounnal and
>
> should list
>
> the articles published, the names and contact addresses of the authors,
> their titles, the names of those who did the peer reviews and required
> disclosers ( such as non independence), weaknesses in the application of
> the IPRASI standards and reasons therefore, etc. and such other data as
> would be pertinent for full and fair disclosure.
>
> the final part of the report should be the Publishers Notes to Articles
> Published Report. The first note might state the nature of the publishing
> company. Its sources of revenues by class of revenues: funds, grants,
> advertising, and so forth, the number of full time and parttime staff and
> the number of different reviewers employed during the year and all such
> else as the IPRASI standards would require. Other notes would be footnoted
> references to articles listed in the Articles Published Report (APR).
>
> Articles Published Report
> XYZ Journal
> Period (Begin Date) to (Enddate)
>
> The major classes of revenues for the XYZ Publishing company listed by
> source is as follows:
> federal grants and funds 5%
> state grants 10%
> Advertising 75%
> Paid for Journal Articles 10%
> and so forth,
> the number of articles that reported on federal subsizied work was 100%.
>
> The number of full time and parttime staff and the number of different
> reviewers employed during the year and all such else as the IPRASI
> standards would require.
>
> Just an idea to account for the differences between the journals and the
> distrust/trust between contributor/publisher.
>
> At 04:28 PM 04/05/2000 +0100, Stevan Harnad wrote:
> >On Wed, 5 Apr 2000, Christopher D. Green wrote:
> >
> >> Although it is well-known among psychologists (it may not be so well-known
> >> among others), there was actually an empirical study of the reliability of
> >> peer review published in Behavioral & Brain Science in 1982. The system did
> >> not come off very well. To quote the abstract:
> >>
> >> "12 research articles were resubmitted to the journals that had published
> >> them 18-32 mo previously, with ficticious names and institutions
> substituted
> >> for the original ones. Only 3 of the resubmissions were detected, and 8 of
> >> the remaining articles were rejected--primarily for "serious methodological
> >> flaws." Authoreviewer accountability is discussed, and recommendations for
> >> improving the peer review system are presented. Commentary on this article
> >> is provided by 56 authors along with the original authors'."
> >>
> >> Peters, Douglas P. & Ceci, Stephen J. (1982). Peer-review practices of
> >> psychological journals: The fate of published articles, submitted again.
> >> Behavioral & Brain Sciences. 1982 Jun Vol 5(2) 187-255
> >
> >Although this is certainly true, and it is also true that peer review could
> >stand with some testing, review, and eventual reform on the basis of the
> outcome,
> >I have to reiterate that the objective of freeing the current
> peer-reviewed literature,
> >such as it is, now, is completely independent of -- and should be kept
> independent of --
> >any future-contingent empirical program for the reform of peer review.
> >
> >----------------------------------------------------------------------
> >Prof. Stevan Harnad bbs_at_cogsci.soton.ac.uk
> >Editor, phone: +44 23-80 594-583
> >Behavioral and Brain Sciences fax: +44 23-80 593-281
> >Department of Electronics and
> > Computer Science http://www.princeton.edu/~harnad/bbs.html
> >University of Southampton http://www.ecs.soton.ac.uk/bbs
> >Highfield, Southampton ftp://ftp.princeton.edu/pub/harnad/BBS
> >SO17 1BJ UNITED KINGDOM
> >
> >
> Computer Aided Cell and Molecular Biology (CACMB), not medicine, will find
> the cure for cancer and other diseases. There will always be a need for
> the trained clinician (MD/RN) but, advanced diagnostic and treatment option
> selection has become gene based, has moved from the physician's practice to
> the computerized cell and molecular biology laboratory, and appropriate
> treatment options should now be based on the personal biology of the
> patient.
>
Received on Mon Jan 24 2000 - 19:17:43 GMT