If I recall correctly, the usability report Les refers to was never published and remains unavailable. So EPrints might be satisfied it has responded to the main findings of that report, but others who are interested in repository design are unable to judge or learn from that.
In terms of the past project reports that may be available, what does all that tell us now? Since repository software is changing and evolving, usability testing has to keep up. It is almost a continuous process. We can built on past evidence, but we must keep it up to date.
It would be surprising if repository software had not iteratively improved the user interface with each new release, given the knowledge and experience of their users. Is that systematically tested for the most recent releases?
Having said that, I still feel the anecdotal evidence from some users that deposit takes to long, etc., points to a more fundamental problem for such users that has more to do with their interest in depositing than in any limitations of the user interface.
Like Les, I am involved with the JISC DepositMO project and I guess we wouldn't be doing this if repository interfaces were truly optimal, even though it is investigating an entirely new and complementary approach.
Steve Hitchcock
IAM Group, Building 32
School of Electronics and Computer Science
University of Southampton, SO17 1BJ, UK
Email: sh94r_at_ecs.soton.ac.uk
Twitter:
http://twitter.com/stevehit
Connotea:
http://www.connotea.org/user/stevehit
Tel: +44 (0)23 8059 7698 Fax: +44 (0)23 8059 2865
On 22 Sep 2010, at 08:12, Leslie Carr wrote:
> I'm just about to start arguing on both sides of the fence :-)
>
> Yes, I agree with these points. UCD is important, and we need to look at the user context, the wider environment in which the user operates, the social norms which contribute to and mitigate against OA, and the plain old usabilty of our software.
>
> And yet, and yet, isn't that exactly what we've been doing for a decade? Certainly a 3-volume independent usability report was one of key components of a fundamental rewrite of our repository software in 2006. Every single one of Tomasz' points have been addressed in repository design and development through (literally) dozens of projects involving a range of end users since 2003. DepositMO (which I mentioned in my last post) is just the latest, but perhaps the most radical, look at how the boundaries of self-deposit can be extended away from the repository itself and integrated onto other parts of a researcher's normal environment (their word processor or their computer desktop).
>
> OA describes itself as a "new public good", so there has been and still has to be a huge amount of adjustment and embedding and attitude shifting and change of practice within the academic community. UCD, by contrast, "tries to optimize the product around how users can, want, or need to use the product, rather than forcing the users to change their behavior to accommodate the product." (quote from wikipedia).
>
> There has to be a balance between OA innovation and UCD. But Tomasz (and other commentators) are right - let's improve the balance with some more user-centredness.
>
> Sent from my iPhone
>
> On 21 Sep 2010, at 22:20, Tomasz Neugebauer <Tomasz.Neugebauer_at_CONCORDIA.CA> wrote:
>
>> I agree, the usability of repository interfaces is inadequately investigated. More generally, the need for more user-centered design methodology in open access advocacy and software design has been apparent to me for some time. User-centered design requires taking into consideration the context of use of technology, and that can be complex in the case of OA repositories: balancing legal/moral obligations (to publishers, institutions, co-authors, funders and the public), digital document version control (pre-print/post-print/publisher version), multimedia attachments, metadata accuracy, web indexing, etc. A computer scientist may have a different context of use from someone working in the humanities - yet the interface has to serve all. In addition to the depositors, the result has to be usable for information seekers (and the tools that they use for research) as well.
>>
>> A user-centered design approach poses questions such as: How efficient and effective are IR interfaces in helping researchers navigate the self-archiving process? How did a change to an IR interface improve efficiency, effectiveness and/or satisfaction of the depositor (and/or information seeker)?
>>
>> I share the opinion that usability of repository interfaces as a broad topic has been inadequately investigated and would like to support user-centered design initiatives.
>>
>>
>> Tomasz Neugebauer
>> Digital Projects & Systems Development Librarian
>> tomasz.neugebauer_at_concordia.ca
>> Concordia University Libraries
>> 1400 de Maisonneuve West (LB 341-3)
>> Tel.: (514) 848-2424 ex. 7738
>>
>>
>>
>> -----Original Message-----
>> From: American Scientist Open Access Forum [mailto:AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On Behalf Of C Oppenheim
>> Sent: Monday, September 20, 2010 10:10 AM
>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>> Subject: Re: Repository effectiveness
>>
>> Steve makes an excellent suggestion for further JISC work. I would be happy to support such an initiative, which should involve experts in usability studies.
>>
>> Charles
>> ________________________________________
>> From: American Scientist Open Access Forum [AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On Behalf Of Steve Hitchcock [sh94r_at_ECS.SOTON.AC.UK]
>> Sent: 20 September 2010 14:10
>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>> Subject: Re: Repository effectiveness
>>
>> The points made by Sally and Charles suggest that the 'why should I bother (to self-archive)?' question is likely to be the primary thought among authors new to open access repositories. This isn't surprising and the effect is easily underestimated in our own enthusiasm. This is the problem addressed by mandates and other initiatives, but clearly there is further to go and this needs continued momentum.
>>
>> It is often convenient or tempting to assume that when a tool or service is not used as widely as expected that this may be something to do with system, software, interface, etc., but this tends to overlook the more fundamental problem of this question above. In fact, it is hard to measure the effectiveness of such aspects unless people are using them properly as intended.
>>
>> Nevertheless, my suspicion is that the usability of repository interfaces as a broad topic has been inadequately investigated and therefore, as also indicated in this thread, there may be weaknesses. A quick scan of Google Scholar reveals some work, but not an extensive list and not all recent. It's not clear if such weaknesses might affect all repositories, some repositories depending on software used, or - since repository interfaces are customisable - individual or local repositories. There may be scope for the current JISC projects on repository deposit, such as DepositMO, to look at this.
>>
>> Steve Hitchcock
>> IAM Group, Building 32
>> School of Electronics and Computer Science
>> University of Southampton, SO17 1BJ, UK
>> Email: sh94r_at_ecs.soton.ac.uk
>> Twitter: http://twitter.com/stevehit
>> Connotea: http://www.connotea.org/user/stevehit
>> Tel: +44 (0)23 8059 7698 Fax: +44 (0)23 8059 2865
>>
>>
>> On 20 Sep 2010, at 12:56, Sally Morris wrote:
>>
>>> I'm not sure Charles is right - certainly, in the study I carried out for
>>> the Bioscience Federation in 2007/8, of 648 who said they did not
>>> self-archive, only 42 said they didn't know how, or had no access to a
>>> repository or support for self-archiving, while a further 23 said they
>>> didn't have time. 'Too difficult' was not mentioned at all
>>>
>>> Sally
>>>
>>>
>>> Sally Morris
>>> South House, The Street, Clapham, Worthing, West Sussex, UK BN13 3UU
>>> Tel: +44 (0)1903 871286
>>> Email: sally_at_morris-assocs.demon.co.uk
>>> -----Original Message-----
>>> From: American Scientist Open Access Forum
>>> [mailto:AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On
>>> Behalf Of C Oppenheim
>>> Sent: 20 September 2010 11:41
>>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>>> Subject: Re: Repository effectiveness
>>>
>>> I am inclined to think it is a combination of the two; on the one hand,
>>> it's not a high priority in the eyes of many researchers; and on the other,
>>> they perceive (wrongly) that it is a chore to self-archive. Indeed, the
>>> idea that it is a chore may be a convenient justification for failing to
>>> take the matter seriously. Having, say, a librarian to take on the job of
>>> doing the self-archiving helps, but doesn't totally overcome some
>>> academics' resistance.
>>>
>>> I also agree that for a mandate to be effective, there must be negative
>>> consequences if the academic does not co-operate.
>>>
>>> Charles
>>> ________________________________________
>>> From: American Scientist Open Access Forum
>>> [AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On Behalf Of
>>> Sally Morris [sally_at_MORRIS-ASSOCS.DEMON.CO.UK]
>>> Sent: 20 September 2010 11:36
>>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>>> Subject: Re: Repository effectiveness
>>>
>>> I am not convinced that the primary obstacle is the difficulty of deposit.
>>> The impression obtained from the studies I did was that the majority of
>>> scholars did not know (or had a very vague and often inaccurate idea) about
>>> self-archiving, and most had no particular interest in depositing their own
>>> work
>>>
>>> A question of mote and beam, perhaps?!
>>>
>>> Sally
>>>
>>>
>>> Sally Morris
>>> South House, The Street, Clapham, Worthing, West Sussex, UK BN13 3UU
>>> Tel: +44 (0)1903 871286
>>> Email: sally_at_morris-assocs.demon.co.uk
>>> -----Original Message-----
>>> From: American Scientist Open Access Forum
>>> [mailto:AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On
>>> Behalf Of Leslie Carr
>>> Sent: 20 September 2010 10:21
>>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>>> Subject: Re: Repository effectiveness
>>>
>>> On 19 Sep 2010, at 16:09, bjork_at_HANKEN.FI wrote:
>>>
>>>> Firstly I have recently uploaded my central 30 articles to our (D-Hanken)
>>> repository,
>>>> In what I would consider best practice fashion. You can check the results
>>> at
>>>> http://www.hanken.fi/staff/bjork/. This took me about one week's workload
>>> in all including finding the proper files, reformatting the personal
>>> versions, checking the copyright issues etc. The actual task of uploading,
>>> once I had everything ready, took perhaps the six minutes suggested, but all
>>> in my experience around an hour would be more appropriate.
>>>
>>> Thanks for providing some actual experience and feedback to the list. I have
>>> had a look at your user record in your institutional DSpace repository, (how
>>> is that related to your home page?, is the material automatically generated
>>> by the repository for inclusion in the home page?) and the 24 items that are
>>> available for public view (perhaps some are stuck in the editorial process?)
>>> appeared at the following times
>>> 3 items on 2010-Apr-28
>>> 5 items on 2010-Jun-01
>>> 8 items on 2010-Jun-17
>>> 5 items on 2010-Aug-12
>>> 3 items on 2010-Aug-16
>>> DSpace does not reveal whether you submitted them in a single batch and the
>>> library processes batched them up, or whether you deposited them in batches
>>> and they were made available immediately.
>>>
>>> I think that the pattern of deposit is important in determining the overall
>>> impact of the workload on the author - and more importantly, on the
>>> psychological impact of the workload. It must be the case that depositing
>>> thirty articles seems like a substantial administrative task, especially
>>> when there are so many other activities demanded of an academic's daily
>>> time. Even five or six items a day is a substantial diary blocker! This is
>>> the backlog phenomenon - any new repository (or new user) has to face the
>>> fact that getting started is the hardest part of using a repository.
>>> Depositing a reasonable representation of your recent (or historical) output
>>> is A Huge Chore. However, once you have achieved that, then the incremental
>>> workload for depositing an individual paper when you have just written it
>>> seems trivial. Especially compared to the job of sorting out the references
>>> :-)
>>>
>>> This was certainly the case for our (school) repository in 2002, when we
>>> decided to mandate the use of EPrints for returning our annual list of
>>> research outputs to the University's admin office. (Stevan may remember
>>> this!) People whined, people complained, people dragged their heels, but
>>> ultimately they did it. But the following year, there were no complaints,
>>> just a few reminders sent out. And an incredibly onerous admin task (a
>>> month's work of 6 staff to produce the departmental research list) was
>>> reduced to a 10 minute job for one person (using Word to reformat the list
>>> that EPrints provided). And since then, we haven't looked back.
>>>
>>> There is a report available which details the study we did at that time to
>>> determine the effort involved in self-deposit:
>>> http://eprints.ecs.soton.ac.uk/10688/
>>> It includes all the data that we collected, and some visualisations of the
>>> Web activity that was involved in depositing several hundred records. That
>>> is where the 6 minute figure comes from, if you are interested.
>>>
>>>> We are helping out some other key researchers at my school to upload and
>>> there are many non-trivial task. For instance researchers in Finance whose
>>> "personal versions" consist of text files and several tables which are
>>> provided to the publishers as sheets in excel files. There may be several
>>> hours of work to format a decent personal version of such a papers. Since
>>> some of best authors are very busy (dean and vice dean of the school) this
>>> has to be done by admin staff.
>>>
>>> You can make a "Sunday best" version of the papers and the spreadsheet
>>> tables, or you could just deposit the texct and the tables separately - if
>>> that is acceptable to the authors. (This is a common phenomenon in Open
>>> Educational Resources - people's teaching materials are never finalised, and
>>> there are always just one or two more adjustments to make to prepare them
>>> for public view. And so a desire for the best sometimes means that material
>>> is never shared.)
>>>
>>>> Secondly the situation reseachers face in making the decision to upload a
>>> green copy resembles the situation faced by any individual deciding whether
>>> or not to take into use a new IT system. There is a large body of literature
>>> on this in Information Systems (my field) research and the UTAUT model :...I
>>> would suggest that using a model like these to model how rational scholars
>>> behave could be could quite fruitful, rather than staring from scratch.
>>>
>>> It would be interesting to analyse some of the Open Access experience from
>>> the last decade in terms of these models, but we are not starting from
>>> scratch in this area. The MIS models are very general, and the OA experience
>>> is very specific. Harnad, for example, maintains a list of 38
>>> rationalisations that people make against the use of repositories:
>>> http://www.eprints.org/openaccess/self-faq/ . Still, adopting an accepted
>>> theoretical framework to talk about this issues can't be a bad thing!
>>>
>>>> Uploading green copies to a repository may not be so different from
>>> starting a profile and uploading stuff to Face Book or other similar
>>> voluntary IT acts we have to decide on.
>>> Except that voluntary participation in Facebook is a million miles away from
>>> formal scholarly communication, in ways that we can all articulate at the
>>> drop of a hat. "Publish or perish" for one!
>>> ---
>>> Les Carr
Received on Wed Sep 22 2010 - 17:02:37 BST