 |
 |
NWDA Reports Phase 1 Round 1Northwest Digital Archives Usability Testing Working Group Report to NWDA Executive Committee
Introduction
The following is the report of the NWDA Usability Testing Working Group. The report summarizes and documents the results of usability testing of the NWDA website and online database, which was conducted between July and October, 2004. A total of 30 respondents contributed to the evaluation.
This report consists of an executive summary and detailed appenixes. The executive summary includes the following sections: Evaluation Overview, Respondents Overview, and Results Summary. Corresponding appendixes provide greater-detailed information on each section, respectively.
Recommendations for improved usability are integrated into the summaries of the page evaluations, and are thereby organized by page and then by evaluation element. Each individual respondent’s ratings and comments are included in the reports in Appendix III, which are also organized by web page and evaluation element.
UTWG
The NWDA Usability Testing Working Group was charged in March 2004 with the following responsibilities:
The NWDA Usability Testing Working Group will develop and implement instruments to evaluate the usability of the NWDA finding aids database by a variety of archival researchers including academic historians, graduate and undergraduate students, genealogists, and institutional researchers. The testing will evaluate: · Search interface: Is it self-explanatory? Are instructions clear? · Search results: Do the search results provide the information necessary for researchers to evaluate the applicability of a given collection to their research? · Browsing Terms: Do browsing terms provide access to significant (but not overwhelming) numbers of collections? · Navigation: Is it clear and straightforward? · Screen design and layout: Does is facilitate research? · Terminology: Do researchers understand the "archival lingo" used on the site (assuming they have received a basic orientation to archival finding aids)? · Researchers' information needs: Does the database provide the information that researchers are seeking about archival and manuscript collections? Usability testing will be conducted during the summer (~June-August 2004). A report of the testing, with recommendations for modifications or enhancements to the NWDA finding aids database, will be completed in September 2004.
Co-chairs of the group were Tony Kurtz and Diana Shenk (Washington State Archives Central and Northwest Regional Branches, respectively). Members of the working group were: Bruce Tabb (University of Oregon), Carolyn McClurkan (Kitsap County, Washington, Historical Society Museum), Cheryl Gunselman (Washington State University), Chris Friday (Western Washington University), Chris Peterson (Oregon State University), Faye Harkins (Oregon State University), Francesca Lane Rasmus (Pacific Lutheran University), Janet Hauk (Whitworth College), Mary Caughey (Oregon State University), Nancy Bunker (Whitworth College), Ron Doel (Oregon State University), and Ryan Wick (Oregon State University). Janet Hauk served as the UTWG liaison to the Executive Committee.
NWDA Usability Testing: Executive Summary
I. Evaluation Overview
Throughout Spring the group designed an evaluation tool. A web-based form was the preferred choice. A draft was created and sited on a Central Washington University web server. The url for the form is http://www.cwu.edu/~archives/NWDAevalForm.htm. The form gathers a profile on each respondent and requests evaluation of the following parts of the NWDA website:
For each of those parts, respondents are requested to score and optionally comment on the effectiveness of each of the following elements:
(See Appendix I for rating criteria for each of the above elements. The criteria are also published for respondents as “rating tips” via links from the online form.)
Respondents are given the opportunity at the end of the form to provide comments on the overall site. Each completed evaluation form is then emailed to Tony Kurtz, who loads the data into an Access database for reporting.
One early set of evaluations took place in mid-July, the remainder in September and early October. The July evaluations were completed as an assignment by 11 students in Chris Friday’s undergraduate Pacific Northwest History course at Western Washington University. The evaluations are reported as a separate unit in the final report primarily because they reflect the online database at a time of early development—a full two months prior to the main evaluation sequence; nevertheless, the students’ comments regarding their assumptions of and experience with the site do serve as valuable feedback from a target user group.
The main evaluation sequence took place between September 1 and October 5. 19 evaluations were completed during that time. The profile of each resondent is included herein as a individual report, with each user identified by a database ID#. The evaluation score and comments of each respondent can therefore be identified in each subsequent report by the presence of the user ID#.
II. Respondents Overview
Respondent profiles are summarized below. The full profile of each respondent is included in Appendix II.
July 2004. 11 students, all unaffiliated with NWDA, completed the evaluation during the July sequence. All were undergraduate students enrolled in an upper-level college course on Pacific Northwest History. All had moderate to low familiarity with their subject area, archival documents, archival finding aids, and online research; but all had, on average, at least moderate experience with online databases.
September - early October 2004. 19 respondents, at least 8 of which were affiliated with NWDA (2 did not check the box “yes” or “no”). A total of 8 were also of the “Archivist-Librarian” user type. Most had at least moderate familiarity with archival documents and archival finding aids, and most had at least moderate experience with online research. All had moderate to high level of experience with online databases.
III. Results Summary
Evaluation results for each element of each page are summarized on the following pages. Full reports are included in Appendix III. Each page summary identifies strengths and makes recommendations for improvements. The numerical rating is on a scale of 1 to 5 with 5 being the highest rating. The following results are from the main evaluation sequence, which took place in September and early October 2004. See section II and Appendix II of this report for profiles of respondents.
Home Page
1. Overall Page Summary
Comments on the home page were generally positive, with a rating between 3.5 and 4.5. The respondents indicated that the functionality and navigation elements worked well. Several concerns about the screen size were expressed.
2. Itemized Summary of Evaluation Elements
Home Page Context
Average rating (5=highest): 4.4
Summary of Comments: Comments were limited but all noted that the home page worked well.
Home Page Design
Average rating (5=highest): 3.5
Summary of comments: Comments reveal some concerns with the home page design, centering on the fact that the page does not fit on most screens. Also several comments on the small font and need for more color and tone contrast. How about an interactive map? Numerous positive comments about the changing photographs, although most want more caption information.
Home Page Functionality
Average rating (5=highest): 4.3
Summary of comments: Comments were limited but mostly positive, as reflected in the average rating. Page functions as expected. One respondent requested the ability to search the site, but not the database. Home Page Information/Language
Average rating (5=highest): 4.1
Summary of comments: Most respondents appreciated the simplicity and brevity of the narrative, although one individual hoped for more interesting and inviting language as the project develops. Also comments on the need for better explanation on how the website could be used and what one should expect to find. Two comments on the need to either delete the Project Description section or provide better explanation about EAD and fining aids. Need for more explanation.
Home Page Navigation
Average rating (5=highest): 4.5
Summary of comments: Again, mostly positive comments. Most noted that the navigation was simple and easy to use. Links were obvious. Same respondent requested ability to search the site, but not the database. A few users could not link to the advanced search page.
Basic Search Page
The majority of comments on the basic search page were positive, praising the clarity and simplicity of the page. Concerns focused mostly on the need for better tools to help the researcher efficiently use Boolean operators and the browsing terms.
Basic Search Page Context
Average rating (5=highest): 4.4
Summary of comments: Very few respondents commented on the page context. Higher rating seems to indicate a basic satisfaction with the way the page fits with the site in general.
Basic Search Page Design
Average rating (5=highest): 3.8
Summary of comments: The positive comments from the respondents focused on the simplicity of the search interface. The lack of clutter made it easy to navigate. The amount of white space around the photograph distracted several users. One suggested that we move the image to the right of the screen and put the search options further to the left. The ability to use Boolean operators in the basic search was cited as a strength but one user felt that this search tool should be more apparent. Users need to click the “help document” link in order to see that Boolean searching is possible. Also, one more negative comment about the need to scroll over to see the whole page.
Basic Search Page Functionality
Average rating (5=highest): 4.1
Summary of comments: Responses reveal a high level of satisfaction with the basic keyword searching. Several of the negative comments focused on the browse list, including expressed confusion about how the keyword search function relates to the browsing terms search. One respondent noted that entering an institution in the basic search and then using the browse list does not limit the search to that one facility.
Basic Search Page Information/Language
Average rating (5=highest): 4.1
ummary of comments: Comments on information/language are similar to the ones made for the functionality of the page. One user felt that the page is straightforward and easy for first time users to understand. Comments again suggest that more clarification is needed for the browse function, including the fact that it is separate from the basic search. The ability to use Boolean operators in the basic search should be clearer.
Basic Search Page Navigation
Average rating (5=highest): 3.9
Summary of comments: One negative response noted that it was very cumbersome for a new window to open with each new finding aid. It’s difficult to navigate a site like NWDA without the functionality of a back button. One user mentioned that her screen froze when she tried to do a “adj2” search. She switched to Netscape and it worked fine. There was also a request for a spelling cross reference check. In addition, several comments on the advanced search page were entered here. These have been summarized under advanced search page navigation.
Advanced Search Page
1. Overall page summary
Several respondents did not find an advanced search page as the advanced search function appears to have been removed from the NWDA site during the evaluation process. In general, the page was rated between 3.1 and 3.7. Comments were positive with respect to the power and functionality of the searches that are possible. Criticism focused on the visual “incompleteness” of the page as well as a range of language and navigational issues that users found difficult or confusing. Given the positive statements regarding the usefulness of a powerful advanced search, it is strongly recommended that the page remain an option, but with the addition of NWDA links, logos, color schemes, and more specific help information regarding access points, such as a “What’s this?” button next to access points. One respondent argued against the use of frames as well.
2. Itemized Summary of Evaluation Elements
Advanced Search Page Context
Average rating (5=highest): 3.7
Summary of comments: Comments regarding shortcomings suggest that the search page did not fit well with the look and feel of the overall NWDA site, which affected usability. Comments regarding strengths focus on the benefits of a powerful search function and that this is consistent with expectations of the NWDA site.
Advanced Search Page Design
Average rating (5=highest): 3.4
Summary of comments: Several respondents noted the visual sparseness of the page and that it appeared to be incomplete. The off-centered look was noted as a weakness, as were the large size and uncoordinated (within NWDA site) color of the buttons. The simplicity of the search function’s design, however, was also cited as a strength.
Advanced Search Page Functionality
Average rating (5=highest): 3.5
Summary of comments: Comments reveal a high level of satisfaction with the advanced search functionality. Only one comment stated a specific drawback: that the number of search options, while a big plus, could also be a deterrent to less-skilled researchers. It appears that the relatively modest average rating of 3.5 is not consistent with the generally positive comments and may reflect other concerns with the page, as noted in other categories.
Advanced Search Page Information/Language
Average rating (5=highest): 3.1
Summary of comments: Many respondents noted confusing language and concepts in the advanced search page. Some of the confusion is related to the use of archival- or web-specific language. One user noted confusion over the identification of levels in the “hits in context” view, another noted confusion over what they would be searching in the “entire finding aid development” field. One respondent suggested strongly that there would be confusion regarding the “sort by” and “order” drop-down boxes, and suggested that they could each be given a default value other than “select criterion”. More than one respondent suggested help screens to inform users specifically what field(s) would be searched using any of the access points offered. In general, comments indicate that this page would benefit from simple help screens that would help users understand the structure of the documents as well as the content of the fields they are searching. A simple suggestion would be a “What’s this?” link next to a search field title.
Advanced Search Page Navigation
Average rating (5=highest): 3.1
Summary of comments: Most respondents noted the absence of the standard NWDA link panel, particularly no link back to the NWDA home page or Basic Search page. Several also stated that it would be confusing to users to have the advanced search page open in a new window, which also affects navigation. On the positive side, one comment praised the pop-up help screens.
Search Results List Page
1. Overall page summary
Overall rating of this page was relatively high, and resondents found it very useful. Specific strong points were the abstracted results for each finding aid and the links into individual finding aids. Of the latter, “Hits in context” was cited several times as a useful tool even if the name and results were a little confusing to some. The link to “xml view” was questioned more than once, and many noted the general lack of links and “look-and-feel” design elements they’d come to expect from the NWDA home page. [It appears that many changes suggested for this page have since been implemented].
2. Itemized Summary of Evaluation Elements
Search Results List Page Context
Average rating (5=highest): 3.9
Summary of comments: Only one substantive comment was offered for this page’s context. That comment expressed the respondent’s disappointment over the results of a search using terms specific to known collections in the database. One other respondent referred to their “Overall Site” comment, which states that the results list page appeared to be out of context with the overall site in look and feel.
Search Results List Page Design
Average rating (5=highest): 4.1
Summary of comments: Overall rating of the page’s design was high, although a few respondents recommended a tighter integration of the design with the NWDA site, specifically in the use of colors and logos or other standard text. One comment specified the size of frame as a drawback—that it was too large. Font color choice of gray instead of black was criticized. [It appears that most of the areas for improvement cited here have been addressed in recent changes to the results page].
Search Results List Page Functionality
Average rating (5=highest): 4.1
Summary of comments: Overall, few comments were offered other than to express a high degree of satisfaction with the functionality of the page. One response praised the links into the finding aids, another stated that “navigation needs help” but that “response time was adequate.”
Search Results Page Information/Language
Average rating (5=highest): 4.2
Summary of comments: The relatively high rating of this page’s functionality was reflected in the positive comments. Respondents praised the usefulness and clear presentation of the finding aid abstracts. “Hits in context” was given high marks as a tool although some expressed a little confusion over its name and how to read the results. The inclusion of “xml markup” as an optional view was questioned. Showing the file size of each finding aid received positive feedback.
Search Results List Navigation
Average rating (5=highest): 4.0
Summary of comments: Respondents’ comments focused primarily on the lack of visible links back to the NWDA home page, etc. “xml markup” was again questioned as to its necessity here. Links into the finding aids were well-received.
Search Results Full Detail
1. Overall page summary
Overall, the individual finding aid details were rated very highly. Consistent praise was offered throughout with respect to the level of detail offered and the clarity of the presentation. Most respondents appeared to be very pleased with the results of their searches. Usability was affected by document size, in particular with respect to navigating back “upward” within a document from within a component level. Others noted that the interface design appeared differently in some browsers, causing text to overlap part of the left-side border. Screen size and the need to scroll were also cited. “Hits in context” was rated highly as a function but also cited as needing improvement with respect to its clarity. This page received some of the best reviews of the entire site.
2. Itemized Summary of Evaluation Elements
Search Results Full Detail Context
Average rating (5=highest): 4.4
Summary of comments: The high rating is supported by consistently positive comments. In general, the finding aids appeared to exceed users’ expectations and were found to be an excellent resource. One respondent stated that all hits were relevant and praised the highlighted hits. Another suggested that there be links from finding aids to a message board where researchers can post comments and share information about specific resources.
Search Results Full Detail Design
Average rating (5=highest): 4.2
Summary of comments: In general, comments were very positive regarding the clarity of the presentation and the ease of understanding the structure of the finding aids. Drawbacks noted include problems with the screen size (scrolling), the choice of colors (“too white and gray”), the light-colored font, and the fact that in one browser setting the gray border overlapped text in the main body of the finding aid. In one case, the use of frames was suggested so that the table of contents links on the left would not disappear.
Search Results Full Detail Functionality
Average rating (5=highest): 4.2
Summary of comments: Very few comments were offered with respect to functionality. The score reflects satisfaction, and the few comments support this. One comment suggested that searches be made available within a single finding aid once retrieved. One respondent noted the problem of being timed out when moving from search results to full detail.
Search Results Full Detail Information/Language
Average rating (5=highest): 3.9
Summary of comments: The rating was affected by one respondent’s score not registering and showing a “0”. Very few comments cited areas needing improvement other than individual encoding errors in finding aids. In general, the language was praised as being thorough, clear, and informative.
Search Results Full Detail Navigation
Average rating (5=highest): 4.0
Summary of comments: Respondents liked the standard links but some found navigation a little difficult as they scrolled down the page and links disappeared. One respondent in particular noted the problem of navigation in large documents and suggested breaking them up; however, another respondent did not find any difficulties navigating the page. One respondent stated that the “hits” link at the top did not work. Another stated that the “hits in context” view was a great idea but found it difficult to determine where the links actually pointed from the way they were stated, which affected navigability. In general, the most common drawback cited was the “vertical” navigation back to the top of a document from within the body.
Overall Site
1. Overall comments summary
Most respondents found the site to be quite useful, with expectations for growth and development. Site contains “excellent data” and “good web design based on simplicity.” Two individuals expressed surprise and satisfaction to find so many related and relevant collections. Strengths also focused on overall design and functionality, with numerous positive comments on changing photographs.
2. Summary of specific concerns
Concerns focused on the following issues:
A. Need for a more seamless movement between pages. Several felt that the overall navigation was clumsy. It was easy to get “lost” between searches.
B. Site designed for academics and scholars. More explanation and navigation tools for amateur researchers.
C. Problems locating advanced search page.
D. Need for expanded definition of “browsing terms.” Respondent questions how comprehensive each of the terms are? How are terms defined and selected? How does it relate to basic search? Why does the list include terms that yield no results when selected for a search? (one user selected “Basque Americans” and did not get any hits).
E. Many computers could not display the entire page on the screen at one time.
Summary of Student Responses from July 2004:
The following paragraphs summarize responses from July 2004. See section II and Appendix II of this report for profiles of respondents.
Students from Chris Friday’s Summer Quarter History 391 course at Western Washington University were given the assignment of visiting the NWDA site, conducting sample searches related to their class research, fill out the online evaluation form, and provide a narrative of their experience using the site. 11 students responded to the survey. Since the web forms were not all filled in completely and contained little commentary (partly due to the state of the web form and partly due to the status of the NWDA site), the students’ narratives are more useful to analyze here. The full texts of their narratives are at the end of Appendix III.
Summary of responses: Students were generally positive about the resource represented by the site. They were impressed with the level of detail found in the finding aids, and most were pleased to see multiple collections relating to their topics. Many also praised the inclusion of collection abstracts in the search result list. The Advanced Search option appears to have been unavailable at the time of evaluation, and several students noted that it would have been a very useful option to use.
Some students were critical of the way the keyword search worked, expressing frustration with the apparent lack of connection between their search terms and the results found. For example, one student did not find an indication of why a particular finding aid met the search criteria. Perhaps a clearer expression of the “Hits in Context” utility may have resolved that difficulty. Other keyword difficulties had as well to do with student expectations of results when entering very specific keywords. Perhaps in this case it may be worth redirecting users to an advanced search page if the keyword search does not meet expectations. |