Copy your main view, switch the new bot filter on in the copy, and compare the two for a few weeks. If the figures make sense, switch it on in the main view and note the date.

A new checkbox in the view settings

Google Analytics added a setting yesterday that removes hits from known bots and spiders. It is a single checkbox headed Bot Filtering in each view's settings, and it starts switched off.

It works on data as it arrives rather than on the code in your pages, so it applies to sites running classic Analytics and Universal Analytics alike.

For owners puzzled by strange visits in their reports, it looks like the answer. It is a useful setting, but it catches less than its name suggests.

What counts as a known bot

Bots and spiders are programs that load web pages automatically: search engine crawlers, services checking whether a site is up, tools copying prices or text, and many others.

Most never run the tracking code, which needs a browser to work, so they never appeared in Analytics at all. A few do run it, and their visits are counted as people.

Every browser and bot sends a short name identifying itself with each request. The new setting compares that name with an industry list of known bots and spiders, and drops the matches.

What the filter does not catch

The setting is a list check, so anything missing from the list, or dishonest about its name, gets through. That leaves several familiar sources of false visits:

The last point surprises people. The setting changes what a view collects from now on, and the months already in your reports stay exactly as they were.

A separate view for testing

A view is one way of looking at a property's data, with its own settings and filters. Most small sites have only the one Analytics created at the start.

A view's settings act on data as it arrives, and nothing can be undone for days already processed. A mistaken setting leaves a permanent dent in the record.

So copy the main view, switch the setting on in the copy, and compare the two over the same weeks. Keep one view untouched as well, as a raw record.

Why the numbers may change

From the day the setting goes on, hits from listed bots disappear from that view. Where there were many, the visits graph drops that day, though no customer behaved differently.

Figures that bots distort can shift too. Bots often load one page and leave, so the bounce rate and the average time on the site may move.

Comparisons across the date mix filtered and unfiltered data, so add an annotation on the day it goes on, in every view, where anyone reading the graphs will see it.

A worked example

A violin maker works alone in a small workshop on the Sunshine Coast, building and repairing violins, violas and cellos for players across Canada and abroad.

The website shows instruments for sale, the repair service and a page of sound recordings. The maker reads Analytics once a month to see which instruments draw attention.

Earlier this summer the reports showed bursts of night-time visits that viewed one page and left, from networks belonging to hosting companies and browsers years out of date.

As soon as the new setting appeared, the maker's developer copied the main view, switched the setting on in the copy, and excluded the workshop's own internet address there.

An annotation marks the date in both views, and an untouched view remains as the raw record. After a few weeks the maker will compare them and decide.

The comparison answers the question either way. If the bursts vanish from the test view, they came from listed bots; if they continue, they are unlisted, and a filter aimed at their pattern can follow.

Reading the comparison

Compare the same days in both views: visits, pages per visit, and the browser and network reports, where bot traffic tends to show as odd versions and hosting companies.

Small differences mean few listed bots were running your tracking code. Large ones mean past reports overstated real visits, which is worth knowing before anyone draws conclusions from them.

Other cleaning worth doing

The bot setting pairs well with two older filters. One excludes the office's own internet address, so staff browsing the site do not count as visitors.

The other accepts only hits recorded on your real domain, which keeps out test copies and pages copied elsewhere with your tracking code still in them.

Both belong in the test view first, for the same reason as the bot setting: a filter can only be judged against a view without it.

Telling the people who read the reports

Anyone who receives the monthly figures, a partner, a bookkeeper or an outside marketer, should hear about the change before they notice it.

A one-line note with the date and the reason saves a meeting spent puzzling over a drop that has nothing to do with customers.

The counter-case

On many small sites the setting will change very little, because few bots that run tracking code ever visit them. An owner who checks the numbers twice a year may never notice.

Any change in how data is collected also makes comparisons across the date less direct, which matters to anyone tracking long trends.

Neither is a reason to leave it off. The setting costs nothing, the test view shows its effect before it touches the main record, and the annotation keeps comparisons honest.

Switching it on

  1. Copy the main view and name the copy as a test.
  2. Tick the bot filtering box in the copy's settings.
  3. Keep one view with no settings or filters at all.
  4. Annotate the date in every view.
  5. Compare the views for three or four weeks.
  6. Switch it on in the main view if the figures make sense.
  7. Tell everyone who reads the reports.

Step three keeps a way back. Every filter is a judgement about what to throw away, and an untouched view is the only record that survives a wrong judgement.

Another source of visits that are not customers, a test copy of the site carrying the live tracking code, is covered in a redesign in progress inflating the visit count.

Frequently asked questions

What does the new bot filtering setting do?

It compares each hit's identifying name with an industry list of known bots and spiders, and leaves the matches out of that view's reports from then on.

Does it remove bots from my past reports?

No. It acts only on data collected after it is switched on. Everything recorded earlier stays as it was, which is why an annotation on that date helps.

Why test it on a separate view first?

Settings act on data as it arrives and cannot be undone afterwards. A copy of the main view shows the effect side by side before it touches your main record.

Will my visit numbers drop?

They may, from the day it is switched on, if listed bots were running your tracking code. A drop then reflects cleaner counting, not a change in customers.

What will the filter miss?

Bots that pretend to be browsers, bots not yet on the list, your own visits, test copies of the site, and anything recorded before it was switched on.

Should every site switch it on?

Most should, after a short test. It costs nothing, and even where it changes little, it removes one source of doubt from the figures.

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.

Strange visits in your Analytics?

Add us as a user on your Analytics account, and we will set up a test view and report what the filter changes.

Start a Conversation