The Googlebot Timeout is 3 Minutes
Googlebot stops crawling pages that take 3 minutes or longer to load. We ran an experiment with a set of…
How to optimize your crawl budget – tips and examples
Let me present an issue that, from my point of view, is crucial to SEO success. Of course, we can…
HTTP & Robots Cheatsheet
This cheatsheet covers every status code and robots directive, and should get you thinking about the impact of using each…
8 Ways of Getting URLs Crawled
Getting URLs Crawled So you already have a website and some of its pages are ranking on Google, great! But…
How to Measure Indexed Pages More Accurately
If the need arises to check how many of a site’s pages are indexed (ie. those URLs that are returned…
Canonical Tags: Easy Dos and Don’ts
So you know about canonical tags. And you know you should be using them. But, with so much information out…
How can non-indexable pages receive Google Organic traffic?
In virtually every Google Analytics view, you’re bound to find landing pages in your Google-Organic report that just shouldn’t be…
Sitemap Audits and Advanced Configuration
While Google and other search engines are getting better at finding pages on their own, Sitemaps can help by giving…
Google times out after two minutes when crawling sitemaps
Our dedicated DeepCrawl engineers are constantly building, testing and pushing SEO to its limits. So it seems only right to…