Before a page can be indexed, and therefore appear within search results, it must be crawled by search engine crawlers, like Googlebot. There are many things to consider in order to get pages crawled and ensure they are adhering to the correct guidelines. These are covered within our Hangout Notes, as well as further research and recommendations.

Google Can Crawl Different Parts of a Website at Different Speeds

September 3, 2019 Source

Google is able to detect how frequently the different sections of a site are updated and crawl them at different speeds, so that the frequently changing pages are crawled more regularly.

Google Determines if Pages Need to be Rendered by Comparing Content Found in Initial HTML & Rendered DOM

August 23, 2019 Source

Google compares the content of the raw HTML of a page from the initial crawl to the rendered DOM after rendering to see if there is new content and to determine if it needs to be rendered going forward.

URL Removal Tool Hides Pages But Doesn’t Impact Crawling or Indexing

August 23, 2019 Source

The URL Removal Tool only hides a page from the search results. Nothing is changed with regards to the crawling and indexing of that page.

Using International IP Redirects Will Prevent Google From Finding Other Versions of A Site

July 12, 2019 Source

If you are redirecting based on international IP addresses, Google is likely to only see the redirect to the English version and would drop all of the other versions.

External Links Are More Critical for Initial Content Discovery & Crawling

June 28, 2019 Source

External links are useful for helping Google find and crawl new websites, but they become less important to Google once it has already discovered the site in question.

Images Implemented Via Lazy Loading Can be Used Like Any Other Image on a Page

June 25, 2019 Source

Images implemented via the lazy load script can be added to structured data and sitemaps without any issues, as long as they are embedded in a way that Googlebot is able to pick up.

Google Doesn’t Need To Be Able To Crawl The Add to Cart Pages of A Site

June 25, 2019 Source

It is not essential for Google to crawl the add to cart pages on e-commerce sites, so this shouldn’t affect a site’s performance in search for purchase intent queries.

Googlebot Does Crawl From a Handful of Regional IPs

June 14, 2019 Source

Googlebot does crawl from a small number of regional IPs, particularly in countries where they know it is hard to crawl from the US.

An Updated User Agent is Expected to Reflect The New Modern Rendering Infrastructure

June 14, 2019 Source

Google has been experimenting with the current user agent settings and is re-thinking the set u. John expects some changes to be announced in the future around an updated user agent so that it reflects the new modern rendering infrastructure.

Related Topics

Indexing Crawl Budget Crawl Errors Crawl Rate Disallow Sitemaps Last Modified Nofollow Noindex RSS Canonicalization Fetch and Render