Design a User Testing Study

The first step to moving forward with your decision to conduct a usability test is thinking through how you want to design it. In this section, you will find information and resources about the basics for developing a user test before pursuing coordination and recruitment activities. This includes forming research questions, developing measurable goals, selecting a methodology, constructing tasks, determining how many test participants are needed, and building a timeline that encompasses the beginning and completion of your study.

Jump to:

Research Questions

Before starting any usability test you need:

  • A reason for what you’re testing
  • Why you’re testing a system with users.

Usability testing research questions should focus on figuring out how users interact with a system, not how the user should adapt to a system’s features. A good approach is to create a research question for what you want to learn from users before jumping into making goals and creating tasks.

If you have a hard time identifying users, the Alliance has library and archive user profiles for you to use on the Persona Resources page, and links to other resources to help you identify an audience on the FAQ/Resources page.

Measurable Goals

After you know what you want to learn from users, the next step is to create measurable goals related to what you want to learn from users. Be aware of what system levels are being assessed: the overall system, a specific user scenario that involves navigating and interaction with various system levels, or focusing on a single page or record. This will help you while shaping your tasks because it determines how many parts of your system have user interactions associated with it.

Make Goals

Goals should be specific, not broad. Broad goals do not allow you to focus on the aim of what you want to learn. For example, “Can users log into the library catalog easily?” is a broad goal. While “Do users notice the button for logging into the library catalog?” is a specific question that emphasizes something that can be assessed.

Make the Goals Measurable

Because “Do users notice the button for logging into the library catalog?” is a specific question that emphasizes something that can be assessed, this is a measurable goal because it can provide quantitative and qualitative metrics for questions like these:

  • How much time does it take a user to notice the button? (Speed)
  • How many attempts does it take before a user notices the button? (Accuracy)
  • What is the overall success in completing the task? (Success)
  • How did the user feel while looking for the login button? (Satisfaction)

Key Points

Things to remember about user testing goals

  • Describe a complete activity
  • Specific and measurable
  • Describe what users want to do, not how they should do that.


How do I create testing goals?

How do I measure a task question outcome?

User Testing Methodologies

There are a number of user testing attitudinal, behavioral, qualitative, qualitative, methodologies to use individually or together on a study. These methods will help you acknowledge and discover what user wants, need, and how you can learn from them in order improve your systems. The goals you make will help identify what one(s) to use.

Questions to consider

  • What do people need? (Behavioral)
  • What do people want? (Attitudinal)
  • Can they use it? (Behavioral)
  • Are you what users saying? (Attitudinal)
  • What are people doing? (Behavioral)
  • Why do they think, say or, do something? (Qualitative)
  • How often or how many times do users do, think, or say something? (Quantitative)

The Methods

When thinking about what methods to use for user research know that there are more than 15 different types to pick from and all have benefits and disadvantages. Six common approaches are highlighted here and whatever ones you use should be based on how to achieve your measurable goals.

  • Interviews and Focus Groups
  • Surveys and Questionnaires
  • A/B Testing
  • Rapid Prototyping
  • Usability Test
  • Card Sorting and Tree Testing


How can I conduct a user test?

How do I know what is the best test to use?

What types of user tests exist?

Task Scenarios

Task scenarios are used to determine what steps are needed to complete a goal. They describe the activity and context with details to complete goals without directing a user on how to complete a task.

Task scenarios should:

  • Give context so users can behave as close to how they would normally complete a task
  • Provide details about what the users should know in order to complete a task
  • Not be prescriptive to the actions you want a user to follow
  • Always have a path to a solution and goal completion


How do I make a task and scenario question?

How do I measure a task question outcome?

Where can I find templates for testing questions?

Test Participants Needed

Ideally, you will need five participants to run a user test. However, it is important to note that if you are testing many different groups, e.g. undergraduates, research faculty, etc., then you need five people from each.


Who are the testing participants?

How many test participants do I need for valid results?

How should I recruit?

What are ethical considerations for user testing?

Make a Timeline

Depending on testing scope, make sure to develop a project timeline before beginning testing. Overall things to consider in your planning stage to help you establish project milestones throughout your process:

  • What time of year are your users available?
  • How long will it take to design the test?
  • How long will it take to perform testing?
  • When will technologies be available for testing?
  • When should all logistics be coordinated and finalized?
  • When will users receive their incentive?
  • When should results be reported?