Google times out after two minutes when crawling sitemaps

Alec Bertram
Alec Bertram

On 24th July 2015 • 2 min read

Our dedicated DeepCrawl engineers are constantly building, testing and pushing SEO to its limits. So it seems only right to share their findings and help you fine-tune your strategy even further. Introducing our new series on SEO experiments: DeepCrawl Labs…

DeepCrawl Labs
 

The test

Sitemaps can sometimes take significant amounts of time to generate, and we wanted to test how Google reacted to sitemaps with differing response times.

 

The method

We set up multiple XML sitemaps with varying response/load delays to see if Google would be patient enough to wait for them. These ranged from 100 to 350 seconds.

 

The results

Sitemaps with a load time of up to 2 minutes/120 seconds were generally accepted without issues. Nothing over 200 seconds was ever indexed.

Sitemaps Load Time

Some slower sitemaps between 120-200 seconds showed a timeout error, but also showed that they were later accepted with submitted and indexed items.

Sitemaps Load Error
 

What this means for you

We noticed that when crawling sitemaps with longer loading times, the crawl speed was negatively affected. We therefore suggest aiming for a 60 second load time for your sitemap to be sure they aren’t causing any problems.

We also recommend using lots of smaller sitemaps, rather than one big one, as this can reduce the sitemap response times.

If your sitemaps are still taking too long to generate, it may be possible to pre-generate them periodically and cache them on the server.

For tips on creating sitemaps, conducting audits or advanced configuration, have a read of this guide.

Author

Alec Bertram
Alec Bertram

Alec is the Head of Product at DeepCrawl. He has 10 years' of experience in SEO, and works to make sure we're building the most valuable things we can for our users.

 

Tags

Get the knowledge and inspiration you need to build a profitable business - straight to your inbox.

Subscribe today