Usability Test Quality Heuristics
The following indicators can help determine if a usability test will produce useful results.
This is not an exhaustive list, but it should be helpful in planning and assessing usability tests. Additions are welcome!
|Researcher clearly articulates the purpose of the usability test (for example, as a specific question to answer, an area of inquiry, etc.).
|Researcher does not specify a purpose for the test, or the purpose specified is very broad, like “testing the app” or “finding problems.”
|Number of participants
|Tests include a sufficient number of participants necessary to see patterns, and there’s only one participant per session.
|Study includes only one participant, or includes more that two participants testing simultaneously (like a focus group).
|Participants are a diverse set of people who actually use the application (or do the task the application is supposed to support).
|Participants are experts (vs. average users), friends, family, or close colleagues of the product team.
|Researcher asks follow up questions that are open-ended and non-leading.
|Questions are leading or subtly suggest potential solutions or hypotheses.
|Researcher asks participants to complete similar tasks related to the study’s purpose.
|Researcher asks participants to complete different tasks, or tasks unrelated to the study’s purpose.
|Researcher asks participants to complete tasks with the product or service that align with their work-related goals.
|Researcher asks participants to complete tasks unrelated to their work-related goals. For example, asking a participant how they might send a fax when their job doesn’t call for that.
|Moderator asks participants to complete tasks without indicating how that might be done. For example, “How would you view the status of your application?”
|Researcher guides participants in completing tasks. For example, “Which of the links in the header would you click to login?”
|Number of teammates
|The team designates a single moderator for the test, and at least one member of the product team observes the usability test.
|A single person from the product team participates in and leads the test.
|Observers do not moderate. They are generally quiet, and ask open-ended questions after the test has concluded.
|Observers interrupt the participant, or attempt to sell or explain the product. Observers debate the participant’s actions in their presence.
|Tests are recorded or notes are taken for sharing with absent stakeholders.
|Tests are not recorded, or test results are not documented.
|Moderator debriefs with teammates immediately after each interview. Researcher looks for patterns across multiple participants and surfaces problems that affected several people.
|Moderator reports the most memorable problems without conducting affinity mapping or some other analysis technique.
|Presentation of findings
|Researcher reports findings to team and stakeholders in an easy to follow, well prioritized way.
|Researcher presents the team a “basket of issues” or an unprioritized laundry list of potential changes.
|Incorporation of findings
|Product team translates findings into future user stories or product refinements.
|Researcher reports do not affect the product backlog or ongoing development work.