DeepCrawl is a website crawler and site architecture analysis tool used by SEOs and web developers which works by crawling web pages.

We obey the Robots.txt protocol and will not crawl your site if you exclude the DeepCrawl user-agent. e.g.

User-agent: deepcrawl

Disallow: /

The IP addresses and user agents used by DeepCrawl will change from time to time depending on user settings as this is sometimes required for specific testing processes.

If you have already excluded the DeepCrawl user-agent but your site is still being crawled without your permission and you would like it to stop, then please contact us on

Find out more with a demo or get started now.

Get a free report now.