Lrileywoods/Usability Testing

Work in Progress
These are Leah's notes on Usability Testing. This is a work in progress and suggestions are welcome. Please use the discussion link above.

What you need to do Usability Testing

• A tester (to run the test)
  • Two Copies of the Consent Form (one for tester, one for participant)
    • this might be covered by the CLA - check with legal?
  • Signup Form
    • This gets information about the participant, as described above
  • A written Test Scenario
  • Paper/laptop for Notes
• A computer (with the application/website you're testing)
• A participant
• A videocamera or screen/audio capture program
• A room where you won't be interrupted during the test

Documents to Create

• Waiver for participants - an outline for a waiver
• Guidelines for creating a usability test
• Guidelines for testers (person running the test)
• Guidelines for participants (person performing the test)
• An example test scenario
• Example test results (a video, some "conclusions" from the test video)
  • http://betterdesktop.org/ has plenty of example data.

About participants

Types of users

• general user
• power user
• sysadmin

Other information to gather on participants

• Primary OS use (Mac, Linux, KDE, Windows, etc.)
• Age
• Gender
• Type of computer user (new to computers, regular, experienced, hacker)
• Experience with Linux (first-time user, casual, experienced)
• Experience with Fedora (first-time user, casual, experienced, dev

team)

If not testing at FUDCon, then get signups, with designation of power user, sys admin, or general user

Things to test

Ideal things to test:

• File Browser
• Software Updater

Authors/maintainers of applications may sign up to volunteer for their application to be tested - they will need to include the information on the application, such as primary function and target participants (general users, power users, sys admins). They can also propose a test scenarios.

Avoid testing things that are not going to change or we are not capable of changing.

Creating a Usability Test

Basic Order

1. Application: Choose an application/website you want to test
2. Intended Use: Determine the intended use of this application/website
3. Intended Audience: Determine the intended audience for the application (general user, power user, sys admin, other)
4. Create a Test Scenario: Create a test scenario for the participant to follow
5. Recruit participants: Five is a good number. You can strong-arm friends and family, recruit online, or just get some co-workers on a slow afternoon
6. Run the Usability Test: Yes, do that.

EXAMPLE:

1. Application: Software Updater
2. Intended Use: Updating the system, which involves downloading and installing new packages
3. Intended Audience: This applies to all users, so general users are the participant choice
4. Test Scenario

Making the Signup form

TODO.

Methods for doing Usability Testing

• Videocamera testing
• Screen Capture & Audio testing (e.g. with istanbul)
• Remote Testing: Screen Capture & Audio testing
• Paper Prototype testing

Usability Testing Protocol

• Thinking Aloud
  • The participant needs to talk through what s/he is doing for the benefit of the recording.
• Active Intervention
  • The tester asks questions, such as "What would you do next?" and "What do you think about the menu structure here?"

Writing a Test Scenario

Write a scenario for the application that tests its function. For example, let's say we're testing the File Browser. The tester would tell the participant to perform a task that would take them through the File Browser.

Example scenario for the File Browser: On this computer, there is a folder titled "Cute Animals" in the users Pictures folder. Go to this folder, and pick your favorite cute animal picture, then move it to the desktop.

Notes about scenario tasks

Generally, a user will be more involved in a task if they get a choice or options to choose from. Of course, for some tasks, this isn't possible, but it's fun to try.

What to do as a Tester

During a test

• Make sure the participant signs the consent form
• Reassure the participant that this is in NO WAY a test of their skills or knowledge, and it is a test of the application itself.
  • This should also be the mindset of the persons evaluating the usability tests.
• Encourage the participant to "think aloud" throughout the test.
  • Sometimes this can be a bit hard, as it's not something people are used to doing. This means ncouraging them to tell you what they're planning to do, why they make the decisions they make, and what they expect to happen when they make a choice. Also, a tester should note a participants' confusion or surprise and the reasons for it.
• You CAN NOT point out where things are or what is happening to the participant - these things should be made clear by the application, and if they aren't, then confusion will be the result of the lone user experience (which is what we're attempting to emulate).
• If a participant is not actually "thinking aloud" very much, ask questions that help get a peek into what they're thinking - for example, if a user hits a button and then says "oh!" or "What?" then ask them what they expected to happen versus what happened.
• Avoid asking the participant leading questions.
  • For example, if a participant is about to go off in the wrong direction, sometimes a testers will ask them "Why are you going to click that?" - a leading question can skew the participant's actions. Try to be more subjective and ask things like "Which of these would you click/select and why?"

After a test

• Summarize the participants experience in notes