** Cross-Posted **
Some prima facie critiques from Steve Hitchcock, concerning the MELIBEA OA Policy Evaluator:
Begin forwarded message:
> From: Steve Hitchcock <sh94r -- ecs.soton.ac.uk>
> Date: July 15, 2010 5:22:30 AM EDT
> To: boai-forum_at_ecs.soton.ac.uk
> Cc: SPARC-OAForum_at_arl.org
>
> Reme, Thank you for bringing this new service to our attention. OA policies are vitally important to the development of institutional repositories, and services that can highlight and bring attention to this development can be valuable.
>
> There are a few aspects of the validation aspects of the new MELIBEA service that confuse, and possibly trouble, me. The first is the main indicator, %OAval, which is the most visible result for a policy. What do you expect this will tell people about a given policy? I randomly selected a couple of policies, one of which was for my own school, to find they each scored about 50%. I would expect these to be among the leaders in terms of OA policies, so this seems a surprisingly unhelpful score.
>
> So what's the explanation? Note that the objects being evaluated are institutional OA policies; they are effectively being presented in relation to institutional repositories when the policy specifies where to archive is an IR with a URL. It seems that the scores include ratings for OA publication policy, libre vs gratis OA, publisher pdf, sanctions (score if Yes), incentives (score if Yes), etc., some of which an institution might specify but which might not apply to an IR http://www.accesoabierto.net/politicas/politicas_estructura.php. However you weight these factors they are still contributors to the overall score, so a policy that is specific to an IR is immediately handicapped, or appears to be unless there is more context to understand the scores.
>
> Which leads me to another question on the visualisation of the validator, and its use of green, gold (and red) in the meter. Do the green and gold refer the the classic OA colours? This would be quite convenient, since it would appear that the green repository policies I mentioned above are achieving almost full scores in the green zone of the meter. However, I suspect this cannot be the case, because it would assume that institutions must have a green AND gold policy, but not simply gold (whatever argument could be put for that).
>
> It is important that new services should help reveal and promote OA policies, as you seek to do, but at the same time not to prejudice the development of such policies by mixing and not fairly separating the contributing factors, especially where these relate to different types of OA.
>
> Steve Hitchcock
> IAM Group, Building 32
> School of Electronics and Computer Science
> University of Southampton, SO17 1BJ, UK
> Email: sh94r -- ecs.soton.ac.uk
> Twitter: http://twitter.com/stevehit
> Connotea: http://www.connotea.org/user/stevehit
> Tel: +44 (0)23 8059 7698 Fax: +44 (0)23 8059 2865
>
> On 15 Jul 2010, at 08:14, Remedios Melero wrote:
>
>> Good mornig!
>> In the last Open Repositories Conference which was held last week in Madrid (http://or2010.fecyt.es/publico/Home/index.aspx ) was presented in the poster session the project called MELIBEA.
>> MELIBEA (http://www.accesoabierto.net/politicas/) is a directory and a validator of institutional open-access (OA) policies regarding scientific and academic work. As a directory, it describes the existing policies. As a validator, it subjects them to qualitative and quantitative analysis based on fulfilment of a set of indicators ( http://www.accesoabierto.net/politicas/politicas_estructura.php) that reflect the bases of an institutional policy.
>>
>> Based on the values assigned to a set of indicators, weighted according to their importance, the validator indicates a score and a percentage of fulfilment for each policy analyzed. The sum of weighted values of each indicator is converted to a percentage scale to give what we have called the “validated open-access percentage” (see how i t is calculated: http://www.accesoabierto.net/politicas/default.php?contenido=acerca ).
>>
>> The types of institution analyzed include universities, research centres, funding agencies and governmental organizations.
>>
>> MELIBEA has three main objectives:
>>
>> • 1. To establish indicators that reveal the strong and weak points of institutional OA polices.
>> • 2. To propose a methodology to guide institutions when they are drawing up an institutional OA policy.
>> • 3. To offer a tool for comparing the contents of policies between institutions.
>> The aim is not to be a ranking, but to offer a tool where to aanlyse and visualize the weaknesses or strenghts of an institutional OA policy based on its wording. It seems something trivial but accomplishment of a policy is based on its terms.
>> Please if you detect any mistake or you would like to make a comment, contact me. I will be pleased if you could check your policy, if any, to analyse our approach.
>> Best wishes
>> Reme
>>
>>
>> R. Melero
>> IATA, CSIC
>> Avda Agustín Escardino 7, 46980 Paterna (Valencia), Spain
>> TEl +34 96 390 00 22. Fax 96 363 63 01
>> E-mail rmelero -- iata.csic.es
>> http://www.accesoabierto.net
Received on Thu Jul 15 2010 - 13:47:30 BST