Northwest Digital Archives
Usability Testing Working Group
Report to NWDA Executive Committee
Introduction
The following is the report
of the NWDA Usability Testing Working Group.
The report summarizes and documents the results of usability testing of
the NWDA website and online database, which was conducted between July and
October, 2004. A total of 30 respondents
contributed to the evaluation.
This report consists of an
executive summary and detailed appenixes.
The executive summary includes the following sections: Evaluation Overview, Respondents Overview,
and Results Summary. Corresponding
appendixes provide greater-detailed information on each section, respectively.
Recommendations for improved
usability are integrated into the summaries of the page evaluations, and are
thereby organized by page and then by evaluation element. Each individual respondent’s ratings and
comments are included in the reports in Appendix III, which are also organized
by web page and evaluation element.
UTWG
The NWDA Usability Testing
Working Group was charged in March 2004 with the following responsibilities:
The NWDA Usability Testing
Working Group will develop and implement instruments to evaluate the usability
of the NWDA finding aids database by a variety of archival researchers
including academic historians, graduate and undergraduate students,
genealogists, and institutional researchers. The testing will evaluate:
·
Search interface:
Is it self-explanatory? Are instructions clear?
·
Search results:
Do the search results provide the information necessary for researchers to
evaluate the applicability of a given collection to their research?
·
Browsing Terms:
Do browsing terms provide access to significant (but not overwhelming) numbers
of collections?
·
Navigation: Is it
clear and straightforward?
·
Screen design and
layout: Does is facilitate research?
·
Terminology: Do
researchers understand the "archival lingo" used on the site
(assuming they have received a basic orientation to archival finding aids)?
·
Researchers' information
needs: Does the database provide the information that researchers are seeking
about archival and manuscript collections?
Usability testing will be
conducted during the summer (~June-August 2004). A report of the testing, with
recommendations for modifications or enhancements to the NWDA finding aids
database, will be completed in September 2004.
Co-chairs of the group were Tony
Kurtz and
NWDA Usability Testing: Executive Summary
I. Evaluation
Overview
Throughout Spring the group
designed an evaluation tool. A web-based
form was the preferred choice. A draft
was created and sited on a
For each of those parts,
respondents are requested to score and optionally comment on the effectiveness
of each of the following elements:
(See Appendix I for rating
criteria for each of the above elements. The criteria are also published for
respondents as “rating tips” via links from the online form.)
Respondents are given the
opportunity at the end of the form to provide comments on the overall
site. Each completed evaluation form is then
emailed to Tony Kurtz, who loads the data into an Access database for reporting.
One early set of evaluations
took place in mid-July, the remainder in September and early October. The July evaluations were completed as an
assignment by 11 students in
The main evaluation sequence
took place between September 1 and October 5.
19 evaluations were completed during that time. The profile of each
resondent is included herein as a individual report, with each user identified
by a database ID#. The evaluation score
and comments of each respondent can therefore be identified in each subsequent
report by the presence of the user ID#.
II. Respondents
Overview
Respondent profiles are
summarized below. The full profile of
each respondent is included in Appendix II.
July
2004.
11
students, all unaffiliated with NWDA, completed the evaluation during the July
sequence. All were undergraduate students enrolled in an upper-level college
course on Pacific Northwest History. All
had moderate to low familiarity with their subject area, archival documents,
archival finding aids, and online research; but all had, on average, at least
moderate experience with online databases.
September
- early October 2004.
19
respondents, at least 8 of which were affiliated with NWDA (2 did not check the
box “yes” or “no”). A total of 8 were
also of the “Archivist-Librarian” user type. Most had at least moderate
familiarity with archival documents and archival finding aids, and most had at
least moderate experience with online research.
All had moderate to high level of experience with online databases.
III. Results
Summary
Evaluation results for each
element of each page are summarized on the following pages. Full reports are included in Appendix III. Each page summary identifies strengths and
makes recommendations for improvements.
The numerical rating is on a scale of 1 to 5 with 5 being the highest
rating.
The following results are
from the main evaluation sequence, which took place in September and early October 2004. See section II and Appendix II of this report
for profiles of respondents.
Home Page
1. Overall Page
Summary
Comments
on the home page were generally positive, with a rating between 3.5 and
4.5. The respondents indicated that the
functionality and navigation elements worked well. Several concerns about the screen size were
expressed.
2. Itemized
Summary of Evaluation Elements
Home Page Context
Average
rating (5=highest): 4.4
Summary
of Comments: Comments were
limited but all noted that the home page worked well.
Home Page Design
Average
rating (5=highest): 3.5
Summary
of comments: Comments reveal some concerns
with the home page design, centering on the fact that the page does not fit on
most screens. Also several comments on
the small font and need for more color and tone contrast. How about an interactive map? Numerous positive comments about the changing
photographs, although most want more caption information.
Home Page Functionality
Average
rating (5=highest): 4.3
Summary
of comments: Comments were limited but
mostly positive, as reflected in the average rating. Page functions as expected. One respondent requested the ability to
search the site, but not the database.
Home Page Information/Language
Average
rating (5=highest): 4.1
Summary
of comments: Most respondents appreciated
the simplicity and brevity of the narrative, although one individual hoped for
more interesting and inviting language as the project develops. Also comments on the need for better
explanation on how the website could be used and what one should expect to
find. Two comments on the need to either
delete the Project Description section or provide better explanation about EAD
and fining aids. Need for more
explanation.
Home Page Navigation
Average
rating (5=highest): 4.5
Summary
of comments: Again, mostly positive
comments. Most noted that the navigation
was simple and easy to use. Links were
obvious. Same respondent requested
ability to search the site, but not the database. A few users could not link to the advanced
search page.
Basic Search Page
The
majority of comments on the basic search page were positive, praising the
clarity and simplicity of the page.
Concerns focused mostly on the need for better tools to help the
researcher efficiently use Boolean operators and the browsing terms.
Basic Search Page Context
Average
rating (5=highest): 4.4
Summary
of comments: Very few respondents
commented on the page context. Higher
rating seems to indicate a basic satisfaction with the way the page fits with
the site in general.
Basic Search Page Design
Average
rating (5=highest): 3.8
Summary
of comments: The positive comments from
the respondents focused on the simplicity of
the search interface. The lack of
clutter made it easy to navigate. The
amount of white space around the photograph distracted several users. One suggested that we move the image to the
right of the screen and put the search options further to the left. The ability to use Boolean operators in the
basic search was cited as a strength but one user felt that this search tool
should be more apparent. Users need to
click the “help document” link in order to see that Boolean searching is
possible. Also, one more negative
comment about the need to scroll over to see the whole page.
Basic Search Page Functionality
Average
rating (5=highest): 4.1
Summary
of comments: Responses reveal a high
level of satisfaction with the basic keyword searching. Several of the negative comments focused on
the browse list, including expressed confusion about how the keyword search
function relates to the browsing terms search. One respondent
noted that entering an institution in the basic search and then using the
browse list does not limit the search to that one facility.
Basic Search Page
Information/Language
Average
rating (5=highest): 4.1
ummary
of comments: Comments on
information/language are similar to the ones made for the functionality of the
page. One user felt that the page is
straightforward and easy for first time users to understand. Comments again suggest that more clarification
is needed for the browse function, including the fact that it is separate from
the basic search. The ability to use
Boolean operators in the basic search should be clearer.
Basic Search Page Navigation
Average
rating (5=highest): 3.9
Summary
of comments: One negative response noted
that it was very cumbersome for a new window to open with each new finding
aid. It’s difficult to navigate a site
like NWDA without the functionality of a back button. One user mentioned that her screen froze when
she tried to do a “adj2” search. She switched to Netscape and it worked
fine. There was also a request for a
spelling cross reference check. In
addition, several comments on the advanced search page were entered here. These have been summarized under advanced
search page navigation.
Advanced Search Page
1. Overall
page summary
Several
respondents did not find an advanced search page as the advanced search
function appears to have been removed from the NWDA site during the evaluation
process. In general, the page was rated between
3.1 and 3.7. Comments were positive with
respect to the power and functionality of the searches that are possible.
Criticism focused on the visual “incompleteness” of the page as well as a range
of language and navigational issues that users found difficult or confusing. Given the positive statements regarding the
usefulness of a powerful advanced search, it is strongly recommended that the
page remain an option, but with the addition of NWDA links, logos, color
schemes, and more specific help information regarding access points, such as a
“What’s this?” button next to access points. One respondent argued against the
use of frames as well.
2. Itemized
Summary of Evaluation Elements
Advanced Search Page Context
Average
rating (5=highest): 3.7
Summary
of comments: Comments regarding
shortcomings suggest that the search page did not fit well with the look and
feel of the overall NWDA site, which affected usability. Comments regarding strengths focus on the
benefits of a powerful search function and that this is consistent with
expectations of the NWDA site.
Advanced Search Page Design
Average
rating (5=highest): 3.4
Summary
of comments: Several respondents noted the
visual sparseness of the page and that it appeared to be incomplete. The off-centered look was noted as a
weakness, as were the large size and uncoordinated (within NWDA site) color of
the buttons. The simplicity of the
search function’s design, however, was also cited as a strength.
Advanced Search Page Functionality
Average
rating (5=highest): 3.5
Summary
of comments: Comments reveal a high level
of satisfaction with the advanced search functionality. Only one comment stated a specific drawback:
that the number of search options, while a big plus, could also be a deterrent
to less-skilled researchers. It appears
that the relatively modest average rating of 3.5 is not consistent with the
generally positive comments and may reflect other concerns with the page, as
noted in other categories.
Advanced Search Page
Information/Language
Average
rating (5=highest): 3.1
Summary
of comments: Many respondents noted
confusing language and concepts in the advanced search page. Some of the
confusion is related to the use of archival- or web-specific language. One user
noted confusion over the identification of levels in the “hits in context”
view, another noted confusion over what they would be searching in the “entire
finding aid development” field. One respondent suggested strongly that there
would be confusion regarding the “sort by” and “order” drop-down boxes, and suggested
that they could each be given a default value other than “select
criterion”. More than one respondent
suggested help screens to inform users specifically what field(s) would be
searched using any of the access points offered. In general, comments indicate that this page
would benefit from simple help screens that would help users understand the structure
of the documents as well as the content of the fields they are searching. A simple suggestion would be a “What’s this?”
link next to a search field title.
Advanced Search Page Navigation
Average
rating (5=highest): 3.1
Summary
of comments: Most respondents noted the
absence of the standard NWDA link panel, particularly no link back to the NWDA
home page or Basic Search page. Several also stated that it would be confusing
to users to have the advanced search page open in a new window, which also
affects navigation. On the positive
side, one comment praised the pop-up help screens.
Search Results List Page
1. Overall
page summary
Overall
rating of this page was relatively high, and resondents found it very
useful. Specific strong points were the
abstracted results for each finding aid and the links into individual finding
aids. Of the latter, “Hits in context”
was cited several times as a useful tool even if the name and results were a
little confusing to some. The link to
“xml view” was questioned more than once, and many noted the general lack of links
and “look-and-feel” design elements they’d come to expect from the NWDA home
page. [It appears that many changes
suggested for this page have since been implemented].
2. Itemized
Summary of Evaluation Elements
Search Results List Page Context
Average
rating (5=highest): 3.9
Summary
of comments: Only one substantive comment
was offered for this page’s context.
That comment expressed the respondent’s disappointment over the results
of a search using terms specific to known collections in the database. One other respondent referred to their
“Overall Site” comment, which states that the results list page appeared to be
out of context with the overall site in look and feel.
Search Results List Page Design
Average
rating (5=highest): 4.1
Summary
of comments: Overall rating of the page’s
design was high, although a few respondents recommended a tighter integration
of the design with the NWDA site, specifically in the use of colors and logos
or other standard text. One comment
specified the size of frame as a drawback—that it was too large. Font color choice of gray instead of black
was criticized. [It appears that most of
the areas for improvement cited here have been addressed in recent changes to
the results page].
Search Results List Page
Functionality
Average
rating (5=highest): 4.1
Summary
of comments: Overall, few comments were
offered other than to express a high degree of satisfaction with the
functionality of the page. One response praised the links into the finding
aids, another stated that “navigation needs help” but that “response time was
adequate.”
Search Results Page
Information/Language
Average
rating (5=highest): 4.2
Summary
of comments: The relatively high rating of
this page’s functionality was reflected in the positive comments. Respondents
praised the usefulness and clear presentation of the finding aid
abstracts. “Hits in context” was given
high marks as a tool although some expressed a little confusion over its name
and how to read the results. The
inclusion of “xml markup” as an optional view was questioned. Showing the file size of each finding aid
received positive feedback.
Search Results List Navigation
Average
rating (5=highest): 4.0
Summary
of comments: Respondents’ comments focused
primarily on the lack of visible links back to the NWDA home page, etc. “xml markup” was again questioned as to its
necessity here. Links into the finding
aids were well-received.
Search Results Full Detail
1. Overall
page summary
Overall,
the individual finding aid details were rated very highly. Consistent praise was offered throughout with
respect to the level of detail offered and the clarity of the
presentation. Most respondents appeared
to be very pleased with the results of their searches. Usability was affected
by document size, in particular with respect to navigating back “upward” within
a document from within a component level.
Others noted that the interface design appeared differently in some
browsers, causing text to overlap part of the left-side border. Screen size and the need to scroll were also
cited. “Hits in context” was rated
highly as a function but also cited as needing improvement with respect to its
clarity. This page received some of the
best reviews of the entire site.
2. Itemized
Summary of Evaluation Elements
Search Results Full Detail Context
Average
rating (5=highest): 4.4
Summary
of comments: The high rating is supported
by consistently positive comments. In
general, the finding aids appeared to exceed users’ expectations and were found
to be an excellent resource. One respondent stated that all hits were relevant
and praised the highlighted hits. Another suggested that there be links from
finding aids to a message board where researchers can post comments and share
information about specific resources.
Search Results Full Detail Design
Average
rating (5=highest): 4.2
Summary
of comments: In general, comments were
very positive regarding the clarity of the presentation and the ease of
understanding the structure of the finding aids. Drawbacks noted include problems with the screen
size (scrolling), the choice of colors (“too white and gray”), the
light-colored font, and the fact that in one browser setting the gray border
overlapped text in the main body of the finding aid. In one case, the use of
frames was suggested so that the table of contents links on the left would not
disappear.
Search Results Full Detail
Functionality
Average
rating (5=highest): 4.2
Summary
of comments: Very few comments were
offered with respect to functionality.
The score reflects satisfaction, and the few comments support this. One comment suggested that searches be made
available within a single finding aid once retrieved. One respondent noted the problem of being
timed out when moving from search results to full detail.
Search Results Full Detail
Information/Language
Average
rating (5=highest): 3.9
Summary
of comments: The rating was affected by
one respondent’s score not registering and showing a “0”. Very few comments cited areas needing
improvement other than individual encoding errors in finding aids. In general, the language was praised as being
thorough, clear, and informative.
Search Results Full Detail Navigation
Average
rating (5=highest): 4.0
Summary
of comments: Respondents liked the
standard links but some found navigation a little difficult as they scrolled
down the page and links disappeared. One
respondent in particular noted the problem of navigation in large documents and
suggested breaking them up; however, another respondent did not find any
difficulties navigating the page. One
respondent stated that the “hits” link at the top did not work. Another stated
that the “hits in context” view was a great idea but found it difficult to
determine where the links actually pointed from the way they were stated, which
affected navigability. In general, the most common drawback cited was the
“vertical” navigation back to the top of a document from within the body.
Overall Site
1.
Overall comments summary
Most
respondents found the site to be quite useful, with expectations for growth and
development. Site contains “excellent
data” and “good web design based on simplicity.” Two individuals expressed surprise and
satisfaction to find so many related and relevant collections. Strengths also focused on overall design and
functionality, with numerous positive comments on changing photographs.
2.
Summary of specific concerns
Concerns
focused on the following issues:
A.
Need for a more
seamless movement between pages. Several
felt that the overall navigation was clumsy.
It was easy to get “lost” between searches.
B.
Site designed for
academics and scholars. More explanation
and navigation tools for amateur
researchers.
C.
Problems locating
advanced search page.
D.
Need for expanded
definition of “browsing terms.”
Respondent questions how comprehensive each of the terms are? How are terms defined and selected? How does it relate to basic search? Why does
the list include terms that yield no results when selected for a search? (one
user selected “Basque Americans” and did not get any hits).
E.
Many computers
could not display the entire page on the screen at one time.
Summary of Student Responses from July 2004:
The following paragraphs
summarize responses from July 2004. See
section II and Appendix II of this report for profiles of respondents.
Students from Chris Friday’s Summer
Quarter History 391 course at
Summary of responses: Students were
generally positive about the resource represented by the site. They were impressed with the level of detail
found in the finding aids, and most were pleased to see multiple collections
relating to their topics. Many also
praised the inclusion of collection abstracts in the search result list. The Advanced Search option appears to have
been unavailable at the time of evaluation, and several students noted that it
would have been a very useful option to use.
Some
students were critical of the way the keyword search worked, expressing
frustration with the apparent lack of connection between their search terms and
the results found. For example, one
student did not find an indication of why a particular finding aid met the
search criteria. Perhaps a clearer
expression of the “Hits in Context” utility may have resolved that difficulty. Other keyword difficulties had as well to do
with student expectations of results when entering very specific keywords. Perhaps in this case it may be worth
redirecting users to an advanced search page if the keyword search does not
meet expectations.