Crawl Budget

Google limits the crawl rate for sites on the same server so that it doesn’t overload the server when crawling these sites.

Google Shopping verification crawling Comes from the overall Crawl Budget

Parameter handling prevents URLs being crawled so is better than canonical tags for crawl efficiency.

You can help to increase your crawl budget by making sure it doesn’t return server errors when Google crawls.

Orphaned pages will be re-crawled, taking up some crawl budget, and usually remain indexed, and can still show up in search results but if there are no internal links then they will be considered unimportant, and they might eventually fall out of the index.

Add nofollow to filtered, sorted and paginated results pages to prevent excessive crawling.

Redirect chains cause latency which can slow down crawling, particularly if there are more than 5 steps which will be rescheduled to be crawled later.

Crawl rate is somewhere between minimum list of pages Google wants to update, and the maximum number of pages they think it’s safe to crawl without impacting performance. Any new pages discovered can be crawled provided there is some remaining budget, but might get queued up for the next day.

If Google discovers a part of your site with a large number of new URLs, it may queue the URLs, generate a Search Console error, but continue to crawl the queued URLs over an extended period.

If you have a ‘reasonably’ sized site (under several thousand pages), you don’t need to worry about crawl budget.