Discovery & Indexing

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus purus elit, tincidunt quis faucibus ut, vehicula sit amet sapien.

Removing Low Quality Pages Takes Months to Impact Crawling and Site Quality

March 20, 2020 Source

Removing low-quality pages from your site may have a positive impact on crawling the rest of the site, but could take 3-9 months until you see changes in crawling which can be measured using log files. Improvements in the overall site quality may take even longer to have an impact. It’s unusual to have any negative impact from removing cruft content.


Use Sitemaps Ping, Last Modified and Separate Sitemaps to Index Updated Content

March 20, 2020 Source

To help Google index updated content more quickly, ping Googlebot when a Sitemap has been updated, use Last Modified dates in Sitemaps, and use a separate Sitemap for updated content so it can be crawled more frequently.


Average Fetch Time May be Affected by Groups of Slower Pages

March 20, 2020 Source

If Google is spending more time crawling a particular group of slow pages then it may make the average fetch time and crawled data look worse.


Rendered Page Resources Are Included in Google’s Crawl Rate

March 20, 2020 Source

The resources that Google fetches when they render a page are included in Google’s crawling budget and reported in the Crawl Stats data in Search Console.


Sites With Slow Response Pages are Crawled Less

March 20, 2020 Source

If Google can’t recrawl pages quickly enough due to poor page response times then they won’t recrawl pages as often as they would like to.


Allow a Single Variation of Category Pages to be Indexed

March 17, 2020 Source

Google doesn’t currently have guidelines on indexing different versions of category pages, but is moving towards recommending allowing a single version to be indexed, such as a sort order, and the alternative variations with different filters and sort orders should be noindexed. If there are other specific versions of category pages which are important, you can allow the first page in the set to be indexed as well.


Google Will Ignore Links on Noindexed Pages Over Time

March 17, 2020 Source

If pages are noindex, Google will ignore those links over time. If you have pages which are only linked from noindex pages then Google may not see the linked pages as important.


Sites Not Indexed in Search May Be Due to Spam or Technical Issues

March 17, 2020 Source

There is a big difference between a site which completely disappears from Google search, and one which is demoted. A site which is removed from the index is usually due to a very significant web spam issue, or a technical issue. If a site is demoted, and not ranking as well as before, it may be due to the quality of the content, or setup of the site. Spammy backlinks are not likely to be a cause.


Search Console Crawl Stats Include URLs Fetched by Other Google Products

March 6, 2020 Source

Google’s crawl stats in Search Console are an accurate reflections of Google’s own crawling logs, but they include URLs fetched from other Google services which use the same infrastructure as Googlebot, including Google Ads landing page checks and product search crawling.


Related Topics

Crawling Indexing Crawl Budget Crawl Errors Crawl Rate Disallow Sitemaps Last Modified Nofollow Noindex RSS Canonicalization Fetch and Render