Depth of Content Affects Crawl Rates
If content is buried deep in the site, it might take longer for Google to discover it, or changes. so improving internal linking from higher levels will help pages be crawled faster.
Crawl Rate is Based on Pages Google Wants to Update
Crawl rate is somewhere between minimum list of pages Google wants to update, and the maximum number of pages they think it’s safe to crawl without impacting performance. Any new pages discovered can be crawled provided there is some remaining budget, but might get queued up for the next day.
Crawl Rate Doesn’t Affect Rankings
Assuming Google is able to pick up your content, there’s no ranking benefit for a page being crawled more frequently.
PDFs Are Crawled Less Frequently Than Pages
PDFs won’t be crawled as often as HTML pages because the content is generally more stable.
Google Checks Robots.txt Once per Day
Most sites have their robots checked once a day, you can speed this up manually by submitting it within the robots.txt tool in Search Console.
Site Traffic Doesn’t Impact Crawl Rate
Traffic volume doesn’t affect crawl frequency, as Google don’t know this. If Google can recognise important pages which frequently contain links to new pages with unique content, they will be crawled more quickly. But crawl rate, or rate of change of a page, doesn’t have any direct relationship to ranking.
Fetch Time Impacts Crawl Rate
Google recommends a page fetch time of less than 1 second, as sites which respond more quickly will get a higher crawl rate. This won’t directly help you rank better but might help get you ranked more quickly.
500 Error Pages May Impact Crawl Rate and Will Eventually be Treated as 404
500 errors can impact ranking. They might result in a lower crawl rate. If they are persistent, they will be treated like a 404 and dropped. Also, Google won’t see the content of a 500 page so you can’t use a meta noindex to get Google to drop those pages if that’s what you want.
Fresh links increase crawling of old pages
If google see’s a new link to an old page, it’s more likely to crawl that page more frequently. So adding new links to re-activated pages should get them discovered more quickly.