I have been following this discussion with interest. I agree with much of what
has been said. I appreciate the UCD efforts that have often been made and at the
same time I appreciate the frustration of those trying to use their local
repository.
As a sociologist who has been a member of multidisciplinary human-centred design
research teams for many years, I see this as a problem that is not only
technical. In my own case when I first tried to enter one of my publications in
our local repository I found it cumbersome and time-consuming. I wondered why
and repeated the exercise with an expert by my side while running Camtasia -
screen capture software. The results showed that the major problems in this case
were social not technical. These included, finding the repository link on the
library home page, gaining access (which at that time required approval); using
language unfamiliar to some scholars, e.g, metadata when all that was required
were the familar elements of a citation. The greatest frustration was that it
was only toward the end of this process that I was asked if I had the rights to
post the article.The latter is a complex issue as we all know, and if uncertain
about the rights, the first choice is likely to 'bail' at that point -- and
probably never to return! So while I think there can always be improvements to
the code and to the UI, we also need to think about ways of improving the
experience by providing advance (brief) information on what will be required.
For example, suggest faculty don't start the process if they are not clear on
the copyright ownership of the piece. Or provide a short video on the steps
required. With subsequent publications, selected to be those for which I knew I
had ownership, the process was very straightforward and extremley fast.
I expect there are other examples of how the first experience/impression could
be improved. I highly recommend taking a future contributor through the process
with the camera running... this simple thinking aloud protocol will not only
reveal the issues, but is engaging and often very funny. And best of all, a
number of the issues may be easily addressed.
Gale Moore, PhD
Senior Fellow & former Director, KMDI (2003-2008)
Member, Graduate Faculty, Dept. of Sociology
Member, Tri-campus Working Group on Scholarly Communication
University of Toronto
On Wed, Sep 22, 2010 at 3:15 PM, Tomasz Neugebauer
<Tomasz.Neugebauer_at_concordia.ca> wrote:
I agree that many improvements to repository software, such as
EPrints, have been motivated, in part, by usability. However, have
all the points really been addressed? How much has repository
software improved over the years in helping depositors to balance
and understand the legal and moral obligations (to publishers,
institutions, co-authors, funders and the public)? How much has
repository software improved over the years in helping depositors to
understand the consequences and implications (in terms of web
indexing) of depositing different versions of their publication?
Moreover, as Steve Hitchcock rightly points out "Since repository
software is changing and evolving, usability testing has to keep
up"; and the results should be published.
More generally, the fact that there is a possible tension between
the following:
1) UCD principle of adjusting/designing/evaluating technology
according to user needs (i.e., human, not AI or search engine)
2) Maximizing OA by creating/mandating a social change in user
behavior through the use of technology and policy.
is a fundamental theoretical problem in OA, in my opinion.
The relevant ISO standard (ISO 13407:1999 Human-centred design
processes for interactive systems) lists the following principles:
* the active involvement of users and a clear understanding of
user and task requirements
* an appropriate allocation of function between users and
technology
* the iteration of design solutions
* multi-disciplinary design.
Is there a tension/conflict between the OA movement (and its new
tools) and any of the above principles? Is what Les calls "OA
Innovation" in conflict with these principles?
-Tomasz
-----Original Message-----
From: American Scientist Open Access Forum
[mailto:AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On
Behalf Of Steve Hitchcock
Sent: Wednesday, September 22, 2010 8:03 AM
To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
Subject: Re: Repository effectiveness
If I recall correctly, the usability report Les refers to was never
published and remains unavailable. So EPrints might be satisfied it has
responded to the main findings of that report, but others who are
interested in repository design are unable to judge or learn from that.
In terms of the past project reports that may be available, what does all
that tell us now? Since repository software is changing and evolving,
usability testing has to keep up. It is almost a continuous process. We
can built on past evidence, but we must keep it up to date.
It would be surprising if repository software had not iteratively improved
the user interface with each new release, given the knowledge and
experience of their users. Is that systematically tested for the most
recent releases?
Having said that, I still feel the anecdotal evidence from some users that
deposit takes to long, etc., points to a more fundamental problem for such
users that has more to do with their interest in depositing than in any
limitations of the user interface.
Like Les, I am involved with the JISC DepositMO project and I guess we
wouldn't be doing this if repository interfaces were truly optimal, even
though it is investigating an entirely new and complementary approach.
Steve Hitchcock
IAM Group, Building 32
School of Electronics and Computer Science
University of Southampton, SO17 1BJ, UK
Email: sh94r_at_ecs.soton.ac.uk
Twitter:
http://twitter.com/stevehit
Connotea:
http://www.connotea.org/user/stevehit
Tel: +44 (0)23 8059 7698 Fax: +44 (0)23 8059 2865
On 22 Sep 2010, at 08:12, Leslie Carr wrote:
> I'm just about to start arguing on both sides of the fence :-)
>
> Yes, I agree with these points. UCD is important, and we need to look at
the user context, the wider environment in which the user operates, the
social norms which contribute to and mitigate against OA, and the plain
old usabilty of our software.
>
> And yet, and yet, isn't that exactly what we've been doing for a decade?
Certainly a 3-volume independent usability report was one of key
components of a fundamental rewrite of our repository software in 2006.
Every single one of Tomasz' points have been addressed in repository
design and development through (literally) dozens of projects involving a
range of end users since 2003. DepositMO (which I mentioned in my last
post) is just the latest, but perhaps the most radical, look at how the
boundaries of self-deposit can be extended away from the repository itself
and integrated onto other parts of a researcher's normal environment
(their word processor or their computer desktop).
>
> OA describes itself as a "new public good", so there has been and still
has to be a huge amount of adjustment and embedding and attitude shifting
and change of practice within the academic community. UCD, by contrast,
"tries to optimize the product around how users can, want, or need to use
the product, rather than forcing the users to change their behavior to
accommodate the product." (quote from wikipedia).
>
> There has to be a balance between OA innovation and UCD. But Tomasz (and
other commentators) are right - let's improve the balance with some more
user-centredness.
>
> Sent from my iPhone
>
> On 21 Sep 2010, at 22:20, Tomasz Neugebauer
<Tomasz.Neugebauer_at_CONCORDIA.CA> wrote:
>
>> I agree, the usability of repository interfaces is inadequately
investigated. More generally, the need for more user-centered design
methodology in open access advocacy and software design has been apparent
to me for some time. User-centered design requires taking into
consideration the context of use of technology, and that can be complex in
the case of OA repositories: balancing legal/moral obligations (to
publishers, institutions, co-authors, funders and the public), digital
document version control (pre-print/post-print/publisher version),
multimedia attachments, metadata accuracy, web indexing, etc. A computer
scientist may have a different context of use from someone working in the
humanities - yet the interface has to serve all. In addition to the
depositors, the result has to be usable for information seekers (and the
tools that they use for research) as well.
>>
>> A user-centered design approach poses questions such as: How efficient
and effective are IR interfaces in helping researchers navigate the
self-archiving process? How did a change to an IR interface improve
efficiency, effectiveness and/or satisfaction of the depositor (and/or
information seeker)?
>>
>> I share the opinion that usability of repository interfaces as a broad
topic has been inadequately investigated and would like to support
user-centered design initiatives.
>>
>>
>> Tomasz Neugebauer
>> Digital Projects & Systems Development Librarian
>> tomasz.neugebauer_at_concordia.ca
>> Concordia University Libraries
>> 1400 de Maisonneuve West (LB 341-3)
>> Tel.: (514) 848-2424 ex. 7738
>>
>>
>>
>> -----Original Message-----
>> From: American Scientist Open Access Forum
[mailto:AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On
Behalf Of C Oppenheim
>> Sent: Monday, September 20, 2010 10:10 AM
>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>> Subject: Re: Repository effectiveness
>>
>> Steve makes an excellent suggestion for further JISC work. I would be
happy to support such an initiative, which should involve experts in
usability studies.
>>
>> Charles
>> ________________________________________
>> From: American Scientist Open Access Forum
[AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On Behalf Of
Steve Hitchcock [sh94r_at_ECS.SOTON.AC.UK]
>> Sent: 20 September 2010 14:10
>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>> Subject: Re: Repository effectiveness
>>
>> The points made by Sally and Charles suggest that the 'why should I
bother (to self-archive)?' question is likely to be the primary thought
among authors new to open access repositories. This isn't surprising and
the effect is easily underestimated in our own enthusiasm. This is the
problem addressed by mandates and other initiatives, but clearly there is
further to go and this needs continued momentum.
>>
>> It is often convenient or tempting to assume that when a tool or
service is not used as widely as expected that this may be something to do
with system, software, interface, etc., but this tends to overlook the
more fundamental problem of this question above. In fact, it is hard to
measure the effectiveness of such aspects unless people are using them
properly as intended.
>>
>> Nevertheless, my suspicion is that the usability of repository
interfaces as a broad topic has been inadequately investigated and
therefore, as also indicated in this thread, there may be weaknesses. A
quick scan of Google Scholar reveals some work, but not an extensive list
and not all recent. It's not clear if such weaknesses might affect all
repositories, some repositories depending on software used, or - since
repository interfaces are customisable - individual or local repositories.
There may be scope for the current JISC projects on repository deposit,
such as DepositMO, to look at this.
>>
>> Steve Hitchcock
>> IAM Group, Building 32
>> School of Electronics and Computer Science
>> University of Southampton, SO17 1BJ, UK
>> Email: sh94r_at_ecs.soton.ac.uk
>> Twitter: http://twitter.com/stevehit
>> Connotea: http://www.connotea.org/user/stevehit
>> Tel: +44 (0)23 8059 7698 Fax: +44 (0)23 8059 2865
>>
>>
>> On 20 Sep 2010, at 12:56, Sally Morris wrote:
>>
>>> I'm not sure Charles is right - certainly, in the study I carried out
for
>>> the Bioscience Federation in 2007/8, of 648 who said they did not
>>> self-archive, only 42 said they didn't know how, or had no access to a
>>> repository or support for self-archiving, while a further 23 said they
>>> didn't have time. 'Too difficult' was not mentioned at all
>>>
>>> Sally
>>>
>>>
>>> Sally Morris
>>> South House, The Street, Clapham, Worthing, West Sussex, UK BN13 3UU
>>> Tel: +44 (0)1903 871286
>>> Email: sally_at_morris-assocs.demon.co.uk
>>> -----Original Message-----
>>> From: American Scientist Open Access Forum
>>> [mailto:AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG]
On
>>> Behalf Of C Oppenheim
>>> Sent: 20 September 2010 11:41
>>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>>> Subject: Re: Repository effectiveness
>>>
>>> I am inclined to think it is a combination of the two; on the one
hand,
>>> it's not a high priority in the eyes of many researchers; and on the
other,
>>> they perceive (wrongly) that it is a chore to self-archive. Indeed,
the
>>> idea that it is a chore may be a convenient justification for failing
to
>>> take the matter seriously. Having, say, a librarian to take on the
job of
>>> doing the self-archiving helps, but doesn't totally overcome some
>>> academics' resistance.
>>>
>>> I also agree that for a mandate to be effective, there must be
negative
>>> consequences if the academic does not co-operate.
>>>
>>> Charles
>>> ________________________________________
>>> From: American Scientist Open Access Forum
>>> [AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG] On
Behalf Of
>>> Sally Morris [sally_at_MORRIS-ASSOCS.DEMON.CO.UK]
>>> Sent: 20 September 2010 11:36
>>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>>> Subject: Re: Repository effectiveness
>>>
>>> I am not convinced that the primary obstacle is the difficulty of
deposit.
>>> The impression obtained from the studies I did was that the majority
of
>>> scholars did not know (or had a very vague and often inaccurate idea)
about
>>> self-archiving, and most had no particular interest in depositing
their own
>>> work
>>>
>>> A question of mote and beam, perhaps?!
>>>
>>> Sally
>>>
>>>
>>> Sally Morris
>>> South House, The Street, Clapham, Worthing, West Sussex, UK BN13 3UU
>>> Tel: +44 (0)1903 871286
>>> Email: sally_at_morris-assocs.demon.co.uk
>>> -----Original Message-----
>>> From: American Scientist Open Access Forum
>>> [mailto:AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG]
On
>>> Behalf Of Leslie Carr
>>> Sent: 20 September 2010 10:21
>>> To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM_at_LISTSERVER.SIGMAXI.ORG
>>> Subject: Re: Repository effectiveness
>>>
>>> On 19 Sep 2010, at 16:09, bjork_at_HANKEN.FI wrote:
>>>
>>>> Firstly I have recently uploaded my central 30 articles to our
(D-Hanken)
>>> repository,
>>>> In what I would consider best practice fashion. You can check the
results
>>> at
>>>> http://www.hanken.fi/staff/bjork/. This took me about one week's
workload
>>> in all including finding the proper files, reformatting the personal
>>> versions, checking the copyright issues etc. The actual task of
uploading,
>>> once I had everything ready, took perhaps the six minutes suggested,
but all
>>> in my experience around an hour would be more appropriate.
>>>
>>> Thanks for providing some actual experience and feedback to the list.
I have
>>> had a look at your user record in your institutional DSpace
repository, (how
>>> is that related to your home page?, is the material automatically
generated
>>> by the repository for inclusion in the home page?) and the 24 items
that are
>>> available for public view (perhaps some are stuck in the editorial
process?)
>>> appeared at the following times
>>> 3 items on 2010-Apr-28
>>> 5 items on 2010-Jun-01
>>> 8 items on 2010-Jun-17
>>> 5 items on 2010-Aug-12
>>> 3 items on 2010-Aug-16
>>> DSpace does not reveal whether you submitted them in a single batch
and the
>>> library processes batched them up, or whether you deposited them in
batches
>>> and they were made available immediately.
>>>
>>> I think that the pattern of deposit is important in determining the
overall
>>> impact of the workload on the author - and more importantly, on the
>>> psychological impact of the workload. It must be the case that
depositing
>>> thirty articles seems like a substantial administrative task,
especially
>>> when there are so many other activities demanded of an academic's
daily
>>> time. Even five or six items a day is a substantial diary blocker!
This is
>>> the backlog phenomenon - any new repository (or new user) has to face
the
>>> fact that getting started is the hardest part of using a repository.
>>> Depositing a reasonable representation of your recent (or historical)
output
>>> is A Huge Chore. However, once you have achieved that, then the
incremental
>>> workload for depositing an individual paper when you have just written
it
>>> seems trivial. Especially compared to the job of sorting out the
references
>>> :-)
>>>
>>> This was certainly the case for our (school) repository in 2002, when
we
>>> decided to mandate the use of EPrints for returning our annual list of
>>> research outputs to the University's admin office. (Stevan may
remember
>>> this!) People whined, people complained, people dragged their heels,
but
>>> ultimately they did it. But the following year, there were no
complaints,
>>> just a few reminders sent out. And an incredibly onerous admin task
(a
>>> month's work of 6 staff to produce the departmental research list) was
>>> reduced to a 10 minute job for one person (using Word to reformat the
list
>>> that EPrints provided). And since then, we haven't looked back.
>>>
>>> There is a report available which details the study we did at that
time to
>>> determine the effort involved in self-deposit:
>>> http://eprints.ecs.soton.ac.uk/10688/
>>> It includes all the data that we collected, and some visualisations of
the
>>> Web activity that was involved in depositing several hundred records.
That
>>> is where the 6 minute figure comes from, if you are interested.
>>>
>>>> We are helping out some other key researchers at my school to upload
and
>>> there are many non-trivial task. For instance researchers in Finance
whose
>>> "personal versions" consist of text files and several tables which are
>>> provided to the publishers as sheets in excel files. There may be
several
>>> hours of work to format a decent personal version of such a papers.
Since
>>> some of best authors are very busy (dean and vice dean of the school)
this
>>> has to be done by admin staff.
>>>
>>> You can make a "Sunday best" version of the papers and the spreadsheet
>>> tables, or you could just deposit the texct and the tables separately
- if
>>> that is acceptable to the authors. (This is a common phenomenon in
Open
>>> Educational Resources - people's teaching materials are never
finalised, and
>>> there are always just one or two more adjustments to make to prepare
them
>>> for public view. And so a desire for the best sometimes means that
material
>>> is never shared.)
>>>
>>>> Secondly the situation reseachers face in making the decision to
upload a
>>> green copy resembles the situation faced by any individual deciding
whether
>>> or not to take into use a new IT system. There is a large body of
literature
>>> on this in Information Systems (my field) research and the UTAUT model
:...I
>>> would suggest that using a model like these to model how rational
scholars
>>> behave could be could quite fruitful, rather than staring from
scratch.
>>>
>>> It would be interesting to analyse some of the Open Access experience
from
>>> the last decade in terms of these models, but we are not starting from
>>> scratch in this area. The MIS models are very general, and the OA
experience
>>> is very specific. Harnad, for example, maintains a list of 38
>>> rationalisations that people make against the use of repositories:
>>> http://www.eprints.org/openaccess/self-faq/ . Still, adopting an
accepted
>>> theoretical framework to talk about this issues can't be a bad thing!
>>>
>>>> Uploading green copies to a repository may not be so different from
>>> starting a profile and uploading stuff to Face Book or other similar
>>> voluntary IT acts we have to decide on.
>>> Except that voluntary participation in Facebook is a million miles
away from
>>> formal scholarly communication, in ways that we can all articulate at
the
>>> drop of a hat. "Publish or perish" for one!
>>> ---
>>> Les Carr
Received on Thu Sep 23 2010 - 04:05:17 BST