A sitemap is a list of all of the live URLs which exist on a site and is used to inform search engine crawlers of the most important pages and therefore which ones should be crawled and indexed. There are several things to consider when creating sitemaps, as well as understanding how search engines view them. We cover a range of these topics within our Hangout Notes, along with best practice recommendations and advice from Google.

Google Measures Sitemap Trust

September 9, 2016 Source

Google has a trust rating per Sitemap, based on incorrect use of last modified data. Google can learn that the last modified date is providing useful information which increases the trust. Otherwise it will start to ignore the last modified dates.

Crawl Errors Priority Metric includes Mixture of Signals

September 9, 2016 Source

The priority metric for crawl errors in search console is a mixture of pages being returned in search results, included in Sitemaps, and if it has internal links. The higher the priority are the ones Google thinks might have content which Google wants to index.

Add Last Modified to Redirects in Sitemaps

August 9, 2016 Source

When Redirecting URLs include them in a Sitemap with a last modified date set after the redirect was put in place, it will encourage them to be crawled more quickly

Last Modified In Sitemaps Aids Crawling

July 8, 2016 Source

Google thinks the Last Modified date in an XML Sitemap can be very useful to help them recrawl URLs, and they also support RSS and Atom feeds.

Google Ignores Irrelevant Sitemap Content

July 8, 2016 Source

Google will ignore any information in Sitemaps which it doesn’t recognise, so you can include additional information for other purposes.

No Good Solution for Reactivating Pages

July 1, 2016 Source

If you have pages which expire but are reactivated after a period of time, there isn’t really a good solution, but you can use a Sitemap to tell Google about URLs which are now active, and use the unavailable-after meta tag.

Mobile Sites Don’t Need Sitemaps

May 17, 2016 Source

Separate mobile sites should be canonicalising to the desktop page, so you don’t need to submit them to Google via a Sitemap, but it’s still worth adding to Search Console.

HTML sitemaps help indexing and crawling

February 26, 2016 Source

If you have a complicated website, providing a mapping of your category pages can help Google to find pages and understand the structure of a website.

Cross Domain Sitemaps Will Be Crawled If Present in Robots.txt

February 23, 2016 Source

Google will use Sitemaps hosted on an external domain if they are referenced in the robots.txt.

Related Topics

Crawling Indexing Crawl Budget Crawl Errors Crawl Rate Disallow Last Modified Nofollow Noindex RSS Canonicalization Fetch and Render