Usability Testing needs more than 5 users – Part 2

Wednesday 14 September 2011 - Filed under Usability News

Now first of all the remote online method as you get if you purchased a Testing User Analysis. The data we received back showed that over the 10 websites tested, when compared to the issues identified the first 5 participants were able to record 29% of the usability issues our experts identified. The 10 and 20 Testing user groups were able to record 46% and 63% respectively.

In the in person observation the results were also inevitable. The User Testing survey questionnaire they had been set concluded that the first five participants recorded 32% of the Usability issues identified and the 10 and 20 Testing User groups were able to record 39% and 69% respectively. Now that’s obviously not completely conclusive as to what you would expect when comparing remote User Testing to controlled lab conditions, but it’s definitely not out of line neither.

We expect to publish the full report shortly but we can draw some preliminary findings.

Firstly Nielsen has suggested that the more users you add to a test, the less you will find and that larger Testing User participant numbers are only really necessary when the site has ‘highly distinct groups of users’. He may be right, but you don’t have to be a Usability Testing expert to know that this data is of course just individual opinions and that the more participants you use the more chance you have of a ‘clearer picture’ as to what your Usability issues actually are.

In his defense I think the purpose of the whole ‘five user theory’ is a pretty innocent attempt to just expand Usability Testing to a wider audience. I mean he critiques expensive lab tests quite strongly but he also suggests that you keep repeating these five person user tests at least three times.

We definitely agree with him that Usability Testing needs to be brought to a wider audience, if that was his intention, but we also think that the suggestion a comprehensive Usability Test can be conducted three times with five people is misguiding. There is only limited choice in the market though. Lets face it from the perspective of a small business you aren’t going to go to the trouble of setting up a controlled lab tests even if it was just 15 people. So you look at remote Usability Testing that’s more obtainable, say Video Testing. Still these cost an average of about $50 so with 15 participants that’s still $750 for some You Tube style videos of 15 users opinions where you have to extract and quantify the data yourself.

Lets not forget for the most part User Testing is still the luxury of large organisations with huge budgets for it. How are we supposed to sell Usability Testing and its benefits to a wider audience when the best the industry can come up with are these inadequate ‘do it yourself’ solutions?

The Testing User analysis was designed to confront the problem head on. Clients can now access the large numbers of participants previously only available for thousands of dollars. From as little as $125 clients can receive valuable insight into the Usability Issues that their web site faces.

The client does not have to make any sacrifices, once you have commissioned the test we do the rest.

We don’t stop there though.

Why should clients who want to conduct Usability& User Testing pay large amounts of money for data they have extract and quantify themselves? The Testing User analysis will arrive to you in a few days as a 30 page board room and presentation ready document outlining the strengths and weaknesses of Usability on your website. This is the direction Usability should be heading

 

Go back to part one of this blog.

2011-09-14  »  admin