Robots.txt Noindex Update: Everything SEOs Need to Know

Read

Common Robots.txt Mistakes and How to Avoid Them

Read

Disallow and Google: An Intermediate SEO Guide

Read

Noindex and Google: A Guide for Intermediate SEOs

Read

Robots Control

Here you’ll find helpful guides on how to create robots.txt files to instruct search engines about websites.

Best Practice

8th July 2019 • 6 min read

Robots.txt Noindex Update: Everything SEOs Need to Know

Update: As of 1st September 2019, Google will be retiring all code that handles unsupported and unpublished rules in robots.txt…

Best Practice

24th January 2019 • 6 min read

Common Robots.txt Mistakes and How to Avoid Them

Robots.txt is a critical tool in an SEO’s arsenal, which is used to establish rules that instruct crawlers and robots…

Best Practice

3rd November 2015 • 13 min read

Disallow and Google: An Intermediate SEO Guide

Following on from our beginner’s guide to implementing noindex, disallow and nofollow directives, we’re now taking a look at some…

Best Practice

28th September 2015 • 9 min read

Noindex and Google: A Guide for Intermediate SEOs

Following on from our beginner’s guide to implementing noindex, disallow and nofollow directives, we’re delving a bit further into the…

Best Practice

17th September 2015 • 16 min read

Noindex, Nofollow & Disallow

The three words above might sound like SEO gobbledegook, but they’re words worth knowing, since understanding how to use them…

Best Practice

26th May 2015 • 12 min read

How to Audit URLs Being Blocked By Robots.txt

When auditing websites that have experienced a drop in rankings and organic search traffic, it’s important to have a good…

Best Practice

17th July 2014 • 8 min read

Managing robots.txt Changes

Writing and making changes to a robots.txt file can make even the most hardened SEOs a little bit nervous. Just…

Get the knowledge and inspiration you need to build a profitable business - straight to your inbox.

Subscribe today