User Testing needs more than five users – Part 1

Wednesday 14 September 2011 - Filed under Usability News

The Usability Testing industry has long stuck to the school of thought that comprehensive User Testing can still be achieved with as little as five users. Lets face it five isn’t much and when dealing with a demographic as widespread as the internet population it just doesn’t seem rational to suggest this is anywhere near enough. This has long been the school of thought though that even with just five Testing Users you could identify over 80% of your websites Usability issues.

We put this to the test with a Testing User study.

To begin with we selected 10 websites of past Testing User clients that varied both in size and nature of content. We selected participants from the Testing User panel in multiples of 5, 10 and 20 to establish what the ratio of improvement was (if any) in the Usability issues identified.

Firstly some background on where this idea that five participants is sufficient for User Testing came from. The first person to suggest this was Nielsen as a conclusion from one of his first studies into User Testing all the way back in 1993. Put simply he suggested that the statistical rigor could be considerably relaxed when conducting real world Usability Testing. By this he means that the Testing User participant numbers can be relaxed as he further went on to suggest in 2000 that elaborate lab controlled Usability Testing were a waste of resources.

This five participant theory has come under fire though. For example one study by Spool and Schroeder found that only 35% of Usability Issues were identified and furthermore they found that both from 13th participant and upwards at least one new Usability issue was discovered. A Usability study shortly after by Perfetti & Landesman discovered that when conducting a study involving 18 participants each new Testing User discovered at least five new Usability issues or obstacles.

 

So we got on the with the study. Now as the Testing User analysis is based on remote methodology, i.e: no laboratory conditions, we conducted the study this way. As is evident from the above 5  person User Testing has been dis proven as a comprehensive method over the years, so we thought it would be more beneficially to compare our ‘remote findings’ with that of previous laboratory tests. Now as the Testing User analysis  differs from traditional usability methodology we decided that there did however need to be some sort of control in place to add validity to our findings. This required us to break away from the Testing User methodology and implement an ‘in person’ test of demographically selected users/participants that we could arrange to ‘come in’ and be observed working their way through the survey.

Our in house Usability Professionals had also identified all the Usability issues that were found after extensive analysis of the 10 websites earlier that month that yet to implement any design changes.

Click here to go to Part 2 of this article: “Usability Testing needs more than 5 users”

2011-09-14  »  admin