Jason Kealey has been writing about software usability (apparently it’s like eggnog). His entire article is worth reading, but I wanted to comment on one specific point he made.
Creating a proper usability study is hard.
- There are so many ways you can introduce bias in a test, it isn’t funny. For this particular test, I feel that it was very close to being unbiased and most of my initial concerns were addressed.
- One concern that does remain is the fact that the test cases were crafted by the researcher and could have been manipulated to generate a specific set of results. However, crafted test cases are required for any kind of comparison between the users. I personally feel the test was unbiased as it presented potential flaws in both the default tool and the proposed tool.
- This particular test was for a tool used by software developers. As you probably know, some developers demonstrate productivity levels an order of magnitude higher than others. You can imagine that this makes it hard to analyze productivity results collected during a study.
These are all very reasonable points, and they do act as potentially confounding variables, if we consider a software usability test as a rigorous scientific experiment. The thing is, I don’t believe we should. In my view, the purpose of usability testing is not to “test” whether a given design is “correct”, but simply to collect enough information to make well-informed decisions regarding the design.
A long time ago, I read an article by the web usability theorist Jakob Nielsen that suggested that the optimum number of users for a usability test is five. Beyond that, you don’t get much more information; you’re just seeing the same bugs all over again.
Software developers work in a world of ones and zeros. On or off. Right or wrong. In code, something either works or it doesn’t. This kind of sharp, detailed-oriented thinking is incredibly valuable when programming, but I don’t think it applies so well to thinking about usability issues or making business decisions. In these cases, you’re not looking to prove something true or false. You’re just looking to get enough data to feel comfortable moving forward with a particular plan.
Don’t misunderstand me; I’m not saying that Jason is wasting his time thinking about usability - quite the opposite, in fact. I just wanted to suggest that worrying about whether a usability test is “unbiased”, or whether your test subjects vary too much in skill level, is probably being unnecessarily rigorous.
The most important things to know about software usability tests is that you should do them, and that you can learn a majority of what you need to by doing the test with just a handful of people. Don’t let your admirably scientific mind overcomplicate this!