Wednesday 19 October 2011 - Filed under Usability News
One questions that gets asks a lot is can you really conduct worthwhile Usability Testing with only five users?
Well there are many different expert opinions concerning the magic number of five. Its important to mention though that User Testing is subjective in the first instance, so saying there is a fixed number of users that can be used in all tests obviously isn’t going to be the one solution.
It is safe to say though that with five users testing it may very well be all you need to answer your questions and uncover the problems concerning your interface.
The theory makes it case by claiming that five users is all that is necessary to uncover approximately 85% of your Usability issues, based on the probability that your Testing users come across a problem as 31%.
Like I said obviously you cant apply this to every User Testing scenario as if the five user theory was used to conduct say a product comparison or task completion rates, it probably wouldn’t work. If you’re looking to uncover the main issues then it will probably be sufficient.
If you think this method might be the best way for you to conduct Usability Testing then your probably also wondering where that 31% discovery rate came from. Well it was an average problem frequency across 4 studies concerning User Testing which discovered that even though not every problem effects 31% of the user population, the most obvious can be detected this way.
I mean if you were conducting Usability Testing on say a released website or piece of software the chances of problem detection per user are closer to 10% or even as low as 5% in some cases. Put simply the more chance there is of there being problems with your product the more Users you will need to test with to detect them. For example if you have a problem that only affects say 10% of you user base, you need to plan for about 18 to have an 85% probability of having that issue reported.
The big question here though concerns how frequently Usability issues actually occur. In the real world we know that a 10% or 31% discovery rate probably don’t really exist, they just represent an average for the problem frequency. This is a sensible assumption, as we know that Usability problems do not consistently affect Users and neither do they do it in a predictable way. The general rule though in User Testing is that if the project is at an early stage then the rate of discovery is likely to be greater than 31%, likewise if the project is already in use and live the rate is likely to be under 10%.
This means that the five user theory does have credibility in that the likelihood a single user will detect a Usability issue is 31%, but you still don’t know what the exact probability is, neither do you know that there is a problem in the first place otherwise you could identify and fix it yourself.
The advice here is to be on the safe side and increase your User Testing panel. If you follow the five Testing User approach then you know that you are likely not to see most of the problems, just the ones that affect 31% and upwards of your users. One the other hand you will probably encounter Usability Issues that effect less than 31% of your Testing User panel.
The Testing User analysis allows for a larger testing panel of up to 150 Users for less than what it would cost to conduct an in person Usability study with five users.
2011-10-19 » admin