Analysing website analytics: ‘Are we drowning in bots?’

My headline question comes following the yearly report from Incapsula, in which they highlight that more than 50% of all online activity last year was, in fact, bot traffic.

These statistics convey that, despite a three-year decline, bot activity is on the rise again – having gone from 61.5% to 56% to 47% to 48.2% in 2013, 2014, 2015 and 2016 respectively.

What is bot traffic?

Bot traffic is artificially generated by machines to register fake visits to your website in tools like Google Analytics. What is there to gain from this? Bot traffic can be made to look like it came from a particular website linking to yours. When you see this site in your analytics reports and visit their website to investigate, they make money from ad revenue.

What does this mean for your website’s analytics?

Simply put, bot traffic skews your web analytics data and obstructs proper analysis and decision making.

Accurate website analytics enables businesses to develop data driven insights and refine their campaigns. Inaccurate, low quality data can lead to misinterpreted activity at every stage of the business. This in turn leads to false assumptions, missed opportunities and potential damage to its site’s performance.

Businesses need to ensure their website reporting and analytics are accurate.

So, where do we start?

Google Analytics is the most dominant and widely used website analytics tool – it makes sense to start there. It has a tremendous 83.4% market share, according to W3Techs and is used by 54.3% of the websites they reviewed.

How can you spot and identify bot traffic in Google Analytics?

Take a look at this screenshot below – see the spike in May 2015?

Image courtesy of The B2B Marketing Lab.

If your website traffic has suddenly spiked unexpectedly – and your business wasn’t hosting any special events or promoting a product or service, it’s likely the spike is due to bot traffic.

Spotting bot traffic in Google Analytics:

1. Firstly, click on Acquisition, then All Traffic and then Channels. Under the Default Channel Grouping column click on Referral.

1. On the Referral list – do you recognise any of the Sources? If not, then it could be bot traffic. Check and cross reference the sources with your brand and clients.

1. What about your metrics? If you have visits showing a 100% bounce rate and an average visit duration of 00:00:00 – you may be looking at bot traffic.

1. You can also click on Secondary Dimension, enter ‘hostname’ and press enter. This will enable you to identify more fake traffic in traffic sources, such as Ghost Referrer spam as this kind of fake traffic almost never uses your websites’ domain or subdomain names as the reported hostname. This is because the bots are targeting many websites at once and do not know your specific domain.

The image below shows an example of ghost referrer spam – the invalid sessions need to be excluded from the reports in order to have a clear view of activity.

Image courtesy of The B2B Marketing Lab.

Using the guidelines above, you can identify part of your spam traffic. However, there are websites online that provide lists of known spammers such as Github and Referrer Spam Blocker.

How can you ignore bot traffic in Google Analytics?

Start by creating a new view in Google Analytics without adding any filters. This will ensure all your original data is saved, removing the risk of losing any data.

There are several actions you can take to free your data from future bot traffic.

Action 1

The first and simplest action you can take is to check the tick-box in the view settings that reads, ‘Exclude all hits from known bots and spiders’. This will filter out bots that Google is aware of.

Action 2

Exclude fake referrals. There are two options for this action, choose the one that works best for you.

Option 1

Create a filter using Google Analytics:

• Click Admin at the bottom of the left-hand side of the Google Analytics dashboard.
• Next, Click the ‘Property’ column, then click on ‘Tracking Info’ then ‘Referral Exclusion List’.
• After clicking on Referral Exclusion List, click: ‘Add Referral Exclusion’.
• Now you can copy and paste the domain names you want to exclude from your Google Analytics. Then click create.
• Repeat the process for all the domains you want to exclude.

This is the quickest way to exclude bot traffic from your Google Analytics, but these filters will only apply to your data moving forward. Your historical data will still contain bots. Be warned that this way you will exclude these referrals from all of your Google Analytics views in that Property so be 100% sure that these referrals are spam before you add them in the exclusion list.

Option 2

Exclude specific referrals within a property:

• Navigate to the Admin area.
• You’ll see three columns, this time click ‘All Filters’ under the ‘Account’ column.
• Click ‘Add Filter’ (you’ll see a red button above the table).
• Next, choose a filter name that’s representative of what you are looking for filter.
• Then click on ‘Custom’ under ‘Filter Type’.
• After clicking on Custom, make sure ‘Exclude’ is selected.
• Next, click on the ‘Filter Field’ drop-down and select ‘Referral’
• In the ‘Filter Pattern’ box, use regular expressions to list the referrer websites that you want to exclude.
• You can then choose to apply this filter to specific views on you Google Analytics, instead of a property as in the first option.
• Click save.

Action 3

This action allows you to get rid of all the Ghost Referrers in your Google Analytics.

• Firstly, make a list of all the domains and subdomains you are tracking in Google Analytics.
• In the Admin area click ‘All Filters’ under ‘Account’.
• Press ‘Add Filter’.
• Choose a filter name, for example: Include only valid hostname.
• Click on ‘Custom’.
• After clicking on Custom, press ‘Include’.
• Once you’ve pressed include, under ‘Filter Field’ select ‘Hostname’.
• Then, in the ‘Filter Pattern’ box, use regular expressions to list the domain names and subdomains that you want to track.
• Apply your filter to the views that you want to use it on, and don’t forget to save!

Ultimately, the methods above will enable you to spot and exclude bot traffic from your Google Analytics. The effectiveness of your reporting, your marketing activities and business decisions are reliant on the quality of your data. Using filters to exclude unwanted activity will keep it clean and ensure you are making decisions based on high quality data.

Read also:

Leverage Google Adwords: techniques to optimise campaigns

Web analytics and safeguarding your site in the ‘age of impersonator bots’