DeepCrawl Bot

DeepCrawl is a website crawler and site architecture analysis tool used by SEOs and web developers which works by crawling web pages.

We obey the Robots.txt protocol and will not crawl your site if you exclude the DeepCrawl user-agent. e.g.

User-agent: deepcrawl
Disallow: /

The IP addresses and user agents used by DeepCrawl will change from time to time depending on user settings as this is sometimes required for specific testing processes.

If you have already excluded the DeepCrawl user-agent but your site is still being crawled without your permission and you would like it to stop, then please contact us on

Find out more with a free trial or get started now.

Start your free trial


Sam Marsden
Sam Marsden

Sam Marsden is DeepCrawl's SEO & Content Manager. Sam speaks regularly at marketing conferences, like SMX and BrightonSEO, and is a contributor to industry publications such as Search Engine Journal and State of Digital.

Get the knowledge and inspiration you need to build a profitable business - straight to your inbox.

Subscribe today