✅ – it’s week 18 of the A to Z of #seo – which means R for…
This is a small file that sits in the root folder of your website. It’s not something a user sees but is something you and I can easily look at as 99.9% of sites have one of these. Just go to the homepage of a site and change your browser/URL by appending robots.txt to the end. (e.g. bbc.co.uk/robots.txt)
This is one of those insignificant-looking files but one that plays a critical role in good SEO as it contains a list of directives which tell each a Search Engine’s ‘bot’ where it can and cannot visit on your site.
As with most things SEO, it can get crazily complicated on big sites (and especially e-commerce sites with 000s of pages) but in the main is fairly straightforward and easy to manipulate.
As ContentKing rightly comments on their guide (link in comments) – “Using the robots file you can prevent search engines from accessing certain parts of your website, prevent duplicate content and give search engines helpful tips on how they can crawl your website more efficiently.
Be careful when making changes to your robots file though: this file has the potential to make big parts of your website inaccessible for search engines.”
One of the big mistakes I often see is when a site launches and the ‘trick’ to hide a dev site from Google is left on by mistake. This looks like:
Deciphered this means, every user-agent (i.e. bot) don’t go anywhere (the / part). You don’t want to do this!
Enjoy and good luck!