Web Data Analysis so far

Friday 17 June 2011 - Filed under Usability News

Despite being the most poplar methodology for Web Data Analysis, Clickstream data is still relatively underused as a tool for enhancing Retail web sites. According to a recent survey near 25% of all internet traffic was accounted for by visits to web retail sites.

Using the power of modern analytics anyone who runs a website can access a stream of detailed data on how those user’s interact with their website. This is generally referred to as Clickstream data. This data can give you a simple overview of site activity or a detailed analysis of visitor clicks and movements and even your own Web servers response times to the page requests.

Most organisations find that despite having Megabytes and Terabytes of this Clickstream data and often a large investment of time and money, few know how to get the most out of these web analytics tools and use the data to their advantage. Put simply if you have a website the chances are you will have access to a server, even your own or rented where you can create a system for the collection of clickstream data by either using a server log either on it’s own or in conjunction with a server plug-in or a Network Pipe. From this your server will ask the visitors computer for information on the computer itself, what the computer is requesting and the date of this request.

Most web site operators use the log format W3C Extended Common Log Format, this method captures quite comprehensive data from simple to date and times to more detailed information like IP address and usernames for sites that require them. Staying with the server logs if you need more detailed data analysis then Plug in’s can be used, mainly for websites like forums and survey sites, which need to monitor data entry and specific events like ‘posts’.

The final server log is the Network Pipe which records the ‘packet data’ being exchanged between two sites. It can record events that wouldn’t get recorder like when you cancel a ‘page load’ or quit a download halfway through. The Network Pipes do allow for some very detailed Clickstream Data as with the other two log’s you could potentially be receiving data on events and page visits which didn’t actually happen.

The Web Analytics market has grown rapidly. Between 2000 and 2005 its was experiencing an 200% year on year growth, and even up to the present it still experiences between 15% – 20% annually. The driving force behind this massive growth has come from that for many e-commerce and retail sites even the prospect of turning a percent or two of browsers into buyers could potentially be worth millions. Still though most web site managers admit they only use web analytic technology for basic web management and feedback.

So what are the reason’s for this underuse?

Well the reasons are complicated but in my opinion is the main issues stem from the amount of analytical methods now available and the complications and problems associated with the analysing of the data itself.

Firstly Clickstream data packages can be huge. In my experience a medium to large website is easily capable of producing over 100 million activity records a day, you also have to take into account all the external data crawlers requesting information on your pages. With this in mind storing and analysing this data can be a really big task.

The second problem concerning the data itself is how disorganised it is. Though all analytics follow a set criteria the way these criteria interact with different browsers brings the data into question. For example there are internet service providers out there who will automatically change the customers IP address or number during a browser session, making it very difficult to determine wether your log is recording single or multiple visits.

The third concern relates to the volume of analytical methods , these include web metric based analysis where you can measure the web sites progress based on effectively marketing buzzwords like slipperiness and stickiness, but these do relate to legitimate needs to measure how well a website attracts and retains it’s visitors. It also relates to more traditional criteria like usability and functionality.

Other web metrics include

- Navigation focussed methods where patterns of page visit are established by using ‘data mining’ techniques, this method relies on you taking the time out to clean your data and categorise your visitors into groups in order to match them with their respective patterns.
- Traffic focussed methods which usually analyse the frequency and times of online store visits which is an important tell of a consumers buying habits.

The evidence so far seems to point to web data analysis being too focussed on the data-capture itself rather than the understanding that is generated when business and organisations focus on Testing the users.

In recent years though the issue of how complicated and sidetracked these web analytics has become has been addressed. The industry of web analytics is moving from server-centric measures over to user-centric measures. The demand has resulted in user testing analysis where the goal has been to develop a structured methodology. Report analysis like the Testing Users experience strive to create as little work as possible for the end user without sacrificing the ability to connect with their user base, presenting accurate data feedback which can easily be applied to benefit their business or organisation.

2011-06-17  »  Sam