Week 18 – robots file

It’s week 18 of the A to Z of #seo – which means R for…

🎯 robots.txt

This is a small file that sits in the root folder of your website. It’s not something a user sees but is something you and I can easily look at as 99.9% of sites have one of these. Just go to the homepage of a site and change your browser/URL by appending robots.txt to the end. (e.g. bbc.co.uk/robots.txt)

This is one of those insignificant-looking files but one that plays a critical role in good SEO as it contains a list of directives which tell each a Search Engine’s ‘bot’ where it can and cannot visit on your site.

As with most things SEO, it can get crazily complicated on big sites (and especially e-commerce sites with 000s of pages) but in the main is fairly straightforward and easy to manipulate.

Years ago, people used to block Javascript (JS) and Styling (CSS) from these ‘bots’ as it was deemed unnecessary but these days Google/Bing HAVE to get to this code in order to crawl and understand your site properly.

As ContentKing rightly comments on their guide (link in comments) – “Using the robots file you can prevent search engines from accessing certain parts of your website, prevent duplicate content and give search engines helpful tips on how they can crawl your website more efficiently.

Be careful when making changes to your robots file though: this file has the potential to make big parts of your website inaccessible for search engines.”

One of the big mistakes I often see is when a site launches and the ‘trick’ to hide a dev site from Google is left on by mistake. This looks like:

User-agent: *
Disallow: /
Deciphered this means, every user-agent (i.e. bot) don’t go anywhere (the / part). You don’t want to do this!

Link to ContentKing guide here – link to Google help file here –  and lastly, Google tool to test one of these files before you make any disastrous changes here.

Enjoy and good luck!