Testing Methods v Testing Users

Sunday 28 August 2011 - Filed under Usability News

There’s no shortage of academics and experts offering their take on what your business needs to do to ensure it has the best web presence possible. My Google experience has shown to me its a fairly crowded market. I mean its not in the same league as would would get back if you searched a term like ‘web design’ but still wherever you look down the results you can choose from a catalogue of study topics written by a variety of individuals or sometimes teams of people who have carried out similar Usability studies from New York all the way to Siberia.

All this of course is at a price and in the name of Market Research, Testing User ‘took the hit’. A hit it was, as the average cost of the 5 reports came out at $189. The cheapest being $148.

In the User Testing market I suppose it’s appropriate to say that this kind of product and methodology is ‘the competition’. I say that with caution as after taking time out to read and reference the methodology against each other I have found that apart from a few advisory similarities they only really have one thing in common. None of them really provide a service.

It’s a bold statement but one I think is true.

I must point out I’m not a Usability Testing fanatic. I am however qualified and very experienced in the field, which I suppose may naturally make me more critical, however I do not  believe that it makes me biased or one-sided, just frustrated.

Frustrated because I was now experiencing the same feeling of disappointment in these products that so many Testing User clients had raised with me. My credit card was now a lot heavier, and with all these different Usability Test methods pointing down different roads, even me, someone  well informed on this specialist subject – was more than confused.

Back to my thought that none of the providers were actually providing a service. This is my perception as yes you could argue that you receive a Usability Testing Video and that definitely ‘legally’ you are receiving a service. Furthermore as you add these User Testing methods to your basket there’s a whole’ load more options available, like for example being able to reproduce the document under license for your employees, for (yes you guessed it) a whole load more money.

I’m not going off subject here but I couldn’t resist pointing out that this website reminded me of a ‘low cost airline’ trying to get me to pay in advance to use the toilet or something.

Back on subject I will say that I don’t think it makes any sense. Of course your always going to get products no matter how specialized that use different methodologies and analysis’s. Usability testing is no different. There lies the problem though.

The Usability methods I bought were all supposed to advise the best methods of making and designing guidelines for a Usability test for e-commerce and retail websites. I ask the question how can products which are supposed to encourage absolute validity in User Testing analysis be so different in opinion? Furthermore how is the $189 average cost justified when effectively you don’t even receive a methodology, just guidelines on ‘how to’ make your own User Testing methodology?

More worryingly for me these reports talked a good game and were very well written and worded, but frustratingly not one stood out as more convincing than the other. As I write this blog I still could not tell you which I would use as my preferred User Analysis out of the 5. At best I could only advise which I think would be more appropriate for a small business over a large business user test.

The final point I will make is that these reports and documents at best point you in the ‘right’ or maybe wrong direction. From looking at the cost and content I don’t think these User Testing products can seriously be aimed at small business. The investments of time and money are too prohibitive.

So these User Testing products must be aimed at large ‘blue chip’ corporations right?

I was contacted recently by a usability engineer contracted for a large online gambling company. He commented on the relevance of these reports and the consultancy of setting up your user study in general.

” The online user analysis and reports I bought are extensive and written well but not realistic in the very real world. Ultimately this is a very expensive DIY solution. The e-commerce user experience report I bought advices that I source clients from different continents, this involves highly controlled lab conditions, international flights that I haven’t even calculated, custom written tests and documents both written and online. It even suggest a heart measuring EMFI chair, which after some research I found isn’t even in production it’s still just a prototype. Many of these user testing report providers offer private consultancy for this well in excess of $40,000. Honestly I think some people in the User Testing market believe large corporations have ‘whatever’ money to spend on this, but we have a budget and want good value just like any company big or small. Testing users is a great service, I bought 10 packages and it gave me a panel of people far greater than what I would have received if I had have gone down the private usability test route. I’m well under budget and Testing Users.com are highly recommended.”

If you ask me this kind of attitude from providers is pretty symptomatic of where the majority of the user testing industry is right now. Over complicated, over priced and inconsistent solutions.

The Testing User’s Experience is meeting these problems head on.

Testing users have invested heavily to create an analysis survey and document that is comprehensive, bespoke, great value and presentation ready. Suitable and available for any business or individual, it really does stand out from the crowd.

Read our Usability Testing – Are the services out their bad? Or just plain Lazy

2011-08-28  »  Sam