Let Google Fetch Your Stylesheets and Scripts

A rule written years ago

Open a small business site's robots.txt, the plain text file at the root of the domain, and there are often rules nobody remembers writing. Some block whole folders: templates, includes, scripts, plugins.

On Monday Google updated its technical guidelines: blocking CSS and JavaScript files directly harms how well its systems render and index a page. A rule that once looked tidy now gets in the way.

Google has been rendering pages much like a browser since the spring, when it asked sites to let it fetch these files. This week the request became part of the guidelines.

Why Google wants the files

A page's HTML holds the words. Its stylesheets decide how they are laid out, and its scripts often build the menu, load the photographs or fill in a list.

Google uses the rendered page to understand what a visitor sees: which text is the main content, which is a menu, and what appears only after a script runs.

If a script that loads part of the content is blocked, that part may as well not exist for Google. The visitor sees it; Googlebot does not.

Rules to look for

Add /robots.txt to the end of your home page address to read the file. These are the Disallow lines that usually cause the trouble:

• Theme or template folders, which hold the stylesheets and often scripts.
• Include and library folders that hold shared scripts.
• Plugin, module or component folders, whose add-ons load their own files.
• Folders named js, css or assets on hand-built sites.
• Patterns ending in .js or .css, written to block those files everywhere.

Some content systems once shipped with rules like these, and many guides recommended adding them to keep Google out of folders that are not pages. Both made sense before Google rendered pages.

Testing a single page

In Webmaster Tools, Fetch as Google has offered Fetch and Render since May. It draws the page as Googlebot saw it, styles and scripts included, ready to set beside the same page in your own browser.

Below the picture it lists the resources Googlebot could not get, with the reason. Where the reason is robots.txt, the fix is in your own file.

The robots.txt Tester, added in July, checks any single address against your rules. Use it to confirm a stylesheet is allowed, before and after an edit.

Editing the file

Remove the Disallow lines that cover stylesheets, scripts and theme folders. Keep the ones that cover admin and login areas, which no visitor needs and which play no part in how public pages look.

Where a folder holds both scripts and things you would rather not have crawled, add Allow lines for the files Google needs, then check each one in the Tester to confirm it is reachable.

Save a copy of the old file first, with the date. If something unexpected changes afterwards, the previous version is there to compare.

Robots.txt is not a lock. A rule there asks well-behaved crawlers to stay out and hides nothing from a person, so keeping the admin area safe is the job of logins and the server.

A worked example

A textile recycling company collects used clothing, shoes and linens from drop-off bins at gas stations, community centres and shopping centre parking lots across the region, then sorts them at a warehouse for resale and recycling.

Its site has a Find a Bin page, where a script loads the bin addresses from a data file and shows them on a map and in a list by town.

The robots.txt, written when the site was built, blocked the scripts and theme folders. Fetch and Render showed the bin page with no layout and no addresses, only a heading above an empty box.

The developer removed both rules, kept the block on the admin area, and checked the data file and the stylesheets in the robots.txt Tester. Fetch and Render now shows the page a visitor sees.

The list of bins also appears in the page as plain text, grouped by town, so the addresses no longer depend on a script at all.

The addresses people search for are now part of what Google renders and reads. A blocked folder had been hiding the page's main content from the one visitor that indexes it.

Files from other sites

Some blocked resources belong to other sites: a map, a font, a sharing button or a booking widget loaded from elsewhere. Their robots.txt is not yours to change.

These matter less when they are decoration. If one carries real content, such as a timetable, make sure the page also says in text what the visitor needs to know.

A map from a mapping service is the common case. The places marked on it should also be written on the page, as the bin company's now are.

Serving the files efficiently

The updated guidelines also ask sites to serve these files efficiently: combined where sensible, compressed, and from a server that can handle Googlebot fetching them as well as visitors.

On shared hosting the server is mostly the host's concern. A page that loads dozens of separate scripts and stylesheets, though, is slow for visitors as much as for Google, and worth fixing either way.

Ask your host whether compression is switched on for stylesheets and scripts. It is a server setting, and many shared plans already have it.

The counter-case

A plain HTML site with one stylesheet and no blocking rules has nothing to change. Google can already fetch everything, and the site may have no robots.txt at all.

Blocking files that play no part in the public page, such as scripts used only in the admin area, does no harm. The guideline is about the files that shape what visitors see.

If you are unsure which is which, test before deleting rules. Fetch and Render shows exactly which blocked files Google wanted for each page.

Clearing the old rules

1. Open your robots.txt in a browser.
2. List every Disallow line and the folder it covers.
3. Run Fetch and Render on your main pages.
4. Note the blocked files it reports.
5. Remove the rules that block stylesheets and scripts.
6. Keep admin areas blocked.
7. Test the changed rules in the robots.txt Tester.

Step three shows whether any of this matters for your site. If Google's picture matches what your own browser shows, the old rules are doing little harm and can be tidied when convenient.

Comparing Google's picture with what your own browser shows, and fixing what differs, are the steps in seeing a page the way Google renders it.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.