Google May Still Crawl Parts of a Site With Desktop Crawler
Even with the shift to mobile-first indexing, Google may still crawl parts of a site with the desktop crawler. John explained that this will not impact the site as long as things are working well on mobile.
Use View Source or Inspect Element to Ensure Hidden Content is Readily Accessible in the HTML
If you have content hidden behind a tab or accordion, John recommends using the view source or inspect element tool to ensure the content is in the HTML by default. Content pre-loaded on the HTML will be treated as normal content on the page, however, if it requires an interaction to load, Google will not be able to crawl or index it.
404 or 410 Status Codes Will Not Impact a Website’s Rankings
If Google identifies 404 or 410 pages on a site, it will continue to crawl these pages in case anything changes, but will begin to phase out the crawling frequency to concentrate more on the pages which return 200 status codes.
Last Modification Dates Important For Recrawling Changed Pages on Large Sites
Including last modification dates on large sites can be important for Google because it helps prioritize the crawling of a changed page which might otherwise take much longer to be recrawled.
Google Has a Separate User Agent For Crawling Sitemaps & For GSC Verification
Google has a separate user agent that fetches the sitemap file, as well as one to crawl for GSC verification. John recommends making sure you are not blocking these.
Blocking Googlebot’s IP is The Best Way to Prevent Google From Crawling Your Site While Allowing Other Tools to Access It
If you want to block Googlebot from crawling a staging site, but want to allow other crawling tools access, John recommends whitelisting the IPs of the users and tools you need to view the site but disallowing Googlebot. This is because Google may crawl pages they find on a site, even if they have a noindex tag, or index pages without crawling them, even if they are blocked in robots.txt.
Ensure Google is Able to Crawl All Pages Involved Within Infinite Scroll
When implementing infinite scroll, ensure Google is able to reach all of the pages involved. John recommends the best way to do this is by linking to all the pages individually through a pagination set up, to ensure each page can be crawled.
Google is Unable to Crawl User-triggered Events
Googlebot cannot crawl user-triggered events, for example content loading once a user scrolls. John recommends using dynamic rendering to enable crawling of these events and ensuring the content loads with a link rather than an interaction.
Prevent Search Engines From Crawling Low Quality UGC
When working with user-generated content, John recommends filtering out high quality pages to ensure search engines are able to see this rather than the lower quality content.