I’m in the middle of a usability study of Fresno State’s Henry Madden Library website.  We’re gearing up to run 42 Fresno State undergrads through ten tasks on the site, from locating a book on a particular subject to finding information about special collections.  The goal is to assess the extent to which the site is useful and effective at delivering access to information.  Can users efficiently use the site to accomplish their tasks?  We know the site does some things well, and that it has problems, but we’re excited to actually explore the depth of the site’s pros and cons with a statistically significant sample.

What’s fascinating and fun about this is the multidisciplinary nature of the project and the constant struggle to clearly define what we are doing.  The team consists of three librarians, Allison Cowgill, Amanda Dinscore and Patrick Newell, plus myself and one of my advanced anthro students, Kim Arnold.  The librarians know the site best, and they know what students need to accomplish on it.  All have prior experience with library- and information-related research.  I’ve found my own best contributions relate to how to handle live human subjects with both respect and a firm guiding hand.  Plus, I think it’s helpful to the project to have non-librarians around who can probe the rationale behind the various questions.  I have found myself asking, repeatedly, “What are you trying to get at when you ask a user to do this?” – forcing an examination of assumptions and goals.  I hope I always have someone asking me those kinds of questions when I’m formulating projects.

Right now, we’re in the testing phase and the instructions to the users and wording of some of the questions are still up in the air.  We’ve tested the protocol on each other, and on the moderators – five anthropology majors who will administer to the test to the actual subjects.  Each time we’ve administered the protocol, we’ve turned up a new issue.  One open issue we have right now is how to handle the well-known student practice of going off site to find resources that are on site.  For example, we know that some students use Google Scholar to search for resources – a practice that sometimes brings them back to the library website itself.  Another example: in a test session, one of the anthro student moderators playing test subject opened a browser page and used Google to find the requested information, and then tracked back to the library website – “found it.”  So, in the test, do we prohibit users from going off site?

Right now, I’m leaning toward prohibition.  We know students use Google and other tools to make the library’s website work for them.  But this study is a test of the library’s website.  If we’re going to gauge how well it works for students, we need to keep them on the site – even if it means short-circuiting their tendency to exit the site and come back from another direction.  If a user struggles to find something, or fails to find it, then we’ll know that’s an area where we need re-examine the site’s architecture.  If we let them go off site to solve the problem, we won’t be able to record the process through which they try to solve the problem using the library site.

This is an odd place for me to come down, since most of the work I do is highly strategic.  I place a high value on trying to see the world from another’s point of view.  In a different context, I’d really want to let the user go to Google and show me how they would really solve the problem.  That’s real user practice and it’s significant.  But in this context, I keep coming back to the goals of the study:  this is a test of the library website, not a study of student search practices.

I can’t wait will our next workgroup meeting where we can hash out this and other issues.  I suspect some of my colleagues will have different approaches to the issues.  Like I said, these collaborative, multidisciplinary projects are FUN.

Comments and opinions are most welcome.