Seeing a Page the Way Google Renders It

A bare page where the design should be

The owner of a small business site tries the new Fetch and Render button in Webmaster Tools this week, and the picture that comes back is not the site the owner knows.

The text is there, black on white, with no colours and no layout. The menu is a plain list of links down the left side, and the product gallery is an empty space.

Open the same page in a browser and it looks exactly as designed. Googlebot and the visitor have been shown two different things.

What Google said, and what it added

Until now, Fetch as Google showed the HTML that Googlebot downloaded and little else. Last Friday, May 23, Google said it now runs a page's scripts and draws it much as a modern browser does.

It asked site owners to let Googlebot fetch the style sheets and scripts each page uses, since a page it cannot draw properly is a page it may misread.

On Tuesday, Fetch and Render arrived in Webmaster Tools, so owners can see the result of that drawing for any page on their site.

What the picture can reveal

On a small site, the differences tend to fall into a few familiar kinds:

• Missing styles, because robots.txt blocks the folder that holds them.
• Menus that never appear, because a script builds them and that script is blocked or fails.
• Empty galleries and maps, filled by scripts or by files on another server.
• Text hidden behind tabs that only a click reveals.
• A desktop layout on the phone fetch, because the styles that adapt the page were never loaded.

Below the picture, the result lists the files Googlebot was not allowed to fetch. That list usually explains most of what the picture shows.

Running it on your own pages

In Webmaster Tools, open Fetch as Google under Crawl, type the address of a page without the domain, and choose Fetch and Render. You can fetch as a desktop browser or as a smartphone.

When the result is ready, open it and look at the picture. Then open the same page in your own browser, side by side, and compare them section by section.

Try the home page, a typical product or service page, and the contact page. Pages built from one template usually share the same problems.

Blocked style sheets and scripts

The commonest cause is a line in robots.txt, often added years ago, that keeps Googlebot out of folders holding the theme, its styles and its scripts.

Those lines were meant to keep system files out of search results. They now also stop Google seeing how the page looks and works.

Removing them is usually a small edit, but robots.txt is easy to get wrong, and a mistyped line can hide the whole site. Leave the admin area blocked, and test the file after every change.

Menus and content built by scripts

Google now runs many scripts, but not necessarily all of them, and not the ones it cannot fetch. Content that appears only after a click or a scroll may never be seen.

A menu written into the HTML works whether or not the script that animates it runs. A menu that a script assembles from scratch depends entirely on that script.

Keep the main links and the key text of each page in the HTML itself, and let scripts add the extras on top.

A worked example

A plumbing fixture showroom in a suburban design district displays taps, sinks, tubs and shower systems for renovators, designers and builders. Its website shows each collection in a scrolling gallery.

When the owner ran Fetch and Render on a collection page, the picture showed a heading, one paragraph and an empty space where the gallery should be. The menu appeared as an unstyled list.

The list of blocked files explained it. The site's robots.txt kept Googlebot out of the folder holding the theme's styles and scripts, a line added when the site was built to tidy the search results.

The gallery was also built entirely by a script reading the products from a separate file, so the page's HTML named no products at all.

The robots.txt line was removed, with the admin area still blocked. Each collection page now carries the product names and short descriptions as text, and the script adds the photographs and scrolling on top.

A second fetch showed the page much as a visitor sees it, with the products listed by name even where a script did not run.

Files from other servers

Fonts, maps, booking widgets and galleries are often loaded from someone else's server, under that server's own robots.txt, and you cannot change those rules.

If the result lists such files as blocked, check whether anything that matters depends on them. A missing font is cosmetic; a product list loaded from elsewhere is not.

When the two pictures match

If Google's picture matches the browser's, nothing needs changing. Fetch the same pages again after any redesign, new theme or change to robots.txt.

Pages that change with each visit, such as a calendar, will differ in detail. What matters is that the layout, the menu and the main text are all present.

The counter-case

A hand-built site with plain HTML, one style sheet and no scripts will usually look the same to Googlebot and to a visitor, and the older fetch already told its owner most of what there was to know.

Google has also said that rendering does not always go perfectly, so a minor difference in a picture is not always worth chasing.

But on a site with a theme, a gallery or a scripted menu, the picture is the quickest way to see what Google sees, and the fixes belong with whoever looks after the site.

Checking your own pages

1. Fetch and render the home page.
2. Compare the picture with the page in your browser.
3. Read the list of blocked files.
4. Open robots.txt and find the lines responsible.
5. Unblock styles and scripts, leaving the admin area closed.
6. Move key text and links into the HTML.
7. Fetch again as a desktop browser and as a smartphone.

Step two is where the problem becomes visible. The blocked list names files, but only the two pictures side by side show what a visitor has and Google lacks.

Another case of Googlebot seeing a different site from the visitor, where a language switch keeps the translated pages out of sight, is described in the French version of the site that Google never shows.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.