** Cross-Posted **
Some prima facie critiques from Steve Hitchcock, concerning the
MELIBEA OA Policy Evaluator:
Begin forwarded message:
From: Steve Hitchcock <sh94r -- ecs.soton.ac.uk>
Date: July 15, 2010 5:22:30 AM EDT
Reme,
Thank you for bringing this new service to our
attention. OA policies are vitally important to the development
of institutional repositories, and services that can highlight
and bring attention to this development can be valuable.
There are a few aspects of the validation aspects of the new
MELIBEA service that confuse, and possibly trouble, me. The first
is the main indicator, %OAval, which is the most visible result
for a policy. What do you expect this will tell people about a
given policy? I randomly selected a couple of policies, one of
which was for my own school, to find they each scored about 50%.
I would expect these to be among the leaders in terms of OA
policies, so this seems a surprisingly unhelpful score.
So what's the explanation? Note that the objects being evaluated
are institutional OA policies; they are effectively being
presented in relation to institutional repositories when the
policy specifies where to archive is an IR with a URL. It seems
that the scores include ratings for OA publication policy, libre
vs gratis OA, publisher pdf, sanctions (score if Yes), incentives
(score if Yes), etc., some of which an institution might specify
but which might not apply to an IR
http://www.accesoabierto.net/politicas/politicas_estructura.php.
However you weight these factors they are still contributors to
the overall score, so a policy that is specific to an IR is
immediately handicapped, or appears to be unless there is more
context to understand the scores.
Which leads me to another question on the visualisation of the
validator, and its use of green, gold (and red) in the meter. Do
the green and gold refer the the classic OA colours? This would
be quite convenient, since it would appear that the green
repository policies I mentioned above are achieving almost full
scores in the green zone of the meter. However, I suspect this
cannot be the case, because it would assume that institutions
must have a green AND gold policy, but not simply gold (whatever
argument could be put for that).
It is important that new services should help reveal and promote
OA policies, as you seek to do, but at the same time not to
prejudice the development of such policies by mixing and not
fairly separating the contributing factors, especially where
these relate to different types of OA.
Steve Hitchcock
IAM Group, Building 32
School of Electronics and Computer Science
University of Southampton, SO17 1BJ, UK
> On 15 Jul 2010, at 08:14, Remedios Melero wrote:
>
> Good mornig!
> In the last Open Repositories Conference which was held last
week in Madrid (
http://or2010.fecyt.es/publico/Home/index.aspx )
was presented in the poster session the project called MELIBEA.
> MELIBEA (http://www.accesoabierto.net/politicas/) is a
directory and a validator of institutional open-access (OA)
policies regarding scientific and academic work. As a directory,
it describes the existing policies. As a validator, it subjects
them to qualitative and quantitative analysis based on fulfilment
of a set of indicators (
http://www.accesoabierto.net/politicas/politicas_estructura.php)
that reflect the bases of an institutional policy.
>
> Based on the values assigned to a set of indicators, weighted
according to their importance, the validator indicates a score
and a percentage of fulfilment for each policy analyzed. The sum
of weighted values of each indicator is converted to a percentage
scale to give what we have called the ?validated open-access
percentage? (see how it is calculated:
http://www.accesoabierto.net/politicas/default.php?contenido=acerca.
>
> The types of institution analyzed include universities,
research centres, funding agencies and governmental
organizations.
>
> MELIBEA has three main objectives:
>
>1. To establish indicators that reveal the strong and weak
points of institutional OA polices.
>2. To propose a methodology to guide institutions when they are
drawing up an institutional OA policy.
>3. To offer a tool for comparing the contents of policies
between institutions.
> The aim is not to be a ranking, but to offer a tool where to
aanlyse and visualize the weaknesses or strenghts of an
institutional OA policy based on its wording. It seems something
trivial but accomplishment of a policy is based on its terms.
> Please if you detect any mistake or you would like to make a
comment, contact me. I will be pleased if you could check your
policy, if any, to analyse our approach.
> Best wishes
> Reme
>
>
> R. Melero
> IATA, CSIC
> Avda Agustin Escardino 7, 46980 Paterna (Valencia), Spain
> TEl +34 96 390 00 22. Fax 96 363 63 01
> E-mail rmelero -- iata.csic.es
> http://www.accesoabierto.net
Received on Fri Jul 16 2010 - 04:04:16 BST