Bot Visits Counted as Customers

Visitors that are not people

Open the visits report for a small business website, and some of the visitors it counts are not people at all.

Programs that crawl the web, check pages, copy content or send spam can behave enough like a browser to be counted as visitors. Some never come near the site and send fake visits straight to the analytics account.

Universal Analytics has a setting that removes the known ones. It stays off unless someone switches it on, and accounts set up before it existed, or in a hurry since, often never did.

So the visits figure is higher than it should be, and every figure calculated from it is wrong in the same direction.

What bots do to the reports

• Visits look higher than the real audience.
• The share who enquire looks lower than it is.
• Bounce rate and time on site are pulled out of shape.
• Some pages look popular only with machines.
• The referral report fills with sites you have never heard of.
• Sudden spikes look like interest that never existed.

The fourth is the one that changes decisions. A business that writes more about whatever seems most read, or promotes whichever page seems busiest, may be taking direction from software.

The setting

In Analytics, go to Admin, choose the view, then View Settings. Under Bot Filtering, tick the box to exclude all hits from known bots and spiders, and save.

The list behind it is maintained for you and updated as new bots are identified. It applies from the moment you save, and earlier data stays as it was.

Each view has its own setting. An account with several views needs the box ticked in every one used for reporting.

Keep one view untouched

Settings and filters in Analytics change data as it is collected, and permanently. A filter that removes too much cannot be undone for the days it ran.

So keep one view with no filters and no bot setting, as a raw copy, and do the reporting in another. If a change ever removes something it should not have, the raw view still has it.

Add an annotation on the day the setting goes on. Later, when a month looks lower than the same month a year earlier, the note will explain why before anyone decides something has gone wrong.

What the setting misses

The list covers bots that announce themselves or are already known. Plenty do neither.

Referral spam is the most visible kind: visits credited to websites with strange names, often offering traffic or free tools, that turn up in the referral report month after month.

Much of it never touches your site. It is sent straight to your tracking ID, which spammers guess or collect, so nothing on the website itself can stop it.

Do not visit those sites to find out what they are. Getting you to click is the whole idea.

Signs of a visit nobody made

A zero-second visit is not suspicious on its own, because Analytics records most one-page visits that way. A source where every single visit looks like that is.

Other signs: a hostname in the network report that is not your own domain, or a browser and screen size listed as not set.

Runs of visits from one city in the middle of the night, all landing on the same page, are another.

The hostname is the most reliable. Real visits happen on your site, or on services you use such as a payment page, so anything else claiming a visit is fake.

A worked example

A bonsai nursery sells trees, pots and tools, runs workshops at weekends, and adds a short care article to its website most weeks.

The owner checked the visits report every Monday and planned new articles, and even some workshops, around whatever had been read most. For months, two articles about carving deadwood and choosing display stands led every report.

No customer had ever mentioned either one. Most of their visits came from websites she had never heard of, or from one distant city overnight, and none went past the first page.

The account had been set up years earlier, before the bot setting existed, and nobody had gone back to tick it.

Her developer ticked it in the main view, kept a raw view untouched, added a filter that accepts only hits for the nursery's own domain, and built a segment to clean up last year's figures.

The articles real visitors read most turned out to be about watering and why junipers struggle indoors, which matched what customers asked at the counter.

Cleaning up last year

Because settings and filters only work forwards, last year's reports still include the bots. A segment can hide them when you look back.

One that includes only your own hostname removes the ghost visits that never touched the site. Excluding the worst referrers by name catches more, though the names change often.

Apply the segment whenever this year is compared with last, or the comparison flatters last year.

Filters for the rest

A hostname filter on the reporting view stops visits that never touched the site from being recorded at all. Test it in a spare view for a week first, since a filter that is wrong discards real data.

Filters that exclude individual spam referrers need adding to as new names appear. They are worth reviewing every few months.

While you are there, a filter for your own office's internet address keeps staff visits out of the figures too. They are not bots, but they are not customers either.

The counter-case

On a site with modest traffic and little spam, the setting changes the totals only slightly, and nobody should expect a dramatic correction.

And Analytics is only one measure. Calls, enquiries and sales are the figures that matter most, and bots do not phone.

Even so, the setting takes a minute and only works from the day it is switched on. Tick it now, keep a raw view, and make every figure from here on closer to the truth.

Before the next report

1. Open View Settings in each reporting view.
2. Tick the bot filtering box and save.
3. Leave one view completely unfiltered.
4. Add an annotation for today's date.
5. Check the hostname report for strangers.
6. Test a hostname filter in a spare view.
7. Build a segment for comparing with last year.

Step four costs ten seconds and saves a confused conversation later. When visits look lower than a year ago, the note says why before anyone concludes the website has stopped working.

Counting the real enquiries, rather than the visits, starts with the enquiries your business lost last year.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.