Sitemaps

A sitemap is a list of all of the live URLs which exist on a site and is used to inform search engine crawlers of the most important pages and therefore which ones should be crawled and indexed. There are several things to consider when creating sitemaps, as well as understanding how search engines view them. We cover a range of these topics within our Hangout Notes, along with best practice recommendations and advice from Google.

Make Sure There is a Clear Connection Between Your Mobile & Desktop Sites

January 9, 2018 Source

It’s possible to include m. pages in your main sitemap file to help Google discover and crawl them for mobile-first, but if there is a clear connection between the desktop and mobile sites then this won’t be necessary.


Canonicals Are Chosen by Google Using XML Sitemap URLs

January 9, 2018 Source

XML sitemap URLs are used to help inform Google’s decision on which URL is chosen to be the canonical.


Crawl Frequency Attribute in XML Sitemaps Doesn’t Impact Crawl Rate

December 15, 2017 Source

Google takes no notice of the crawl frequency attribute in XML sitemaps or any priority set. Only the last modification timestamp will impact crawl rate.


Google Uses Scheduler to Determine Recrawl Date

December 15, 2017 Source

Google uses a scheduler before crawling to work out when they need to recrawl URLs. Google will increase crawl rate if it gets signals that it needs to do so e.g. updated modification date in sitemaps and internal linking (especially from the homepage).


New Search Console Will Show More Sitemap Data

December 12, 2017 Source

The new Search Console will show more detailed information regarding sitemaps and more detail per sitemap file.


Use Several Smaller Sitemaps to Locate Indexing Issues

December 12, 2017 Source

Having several smaller sitemaps for the different sections of your site is recommended for diagnosing indexing issues.


URLs in Sitemaps Are Not Guaranteed to be Indexed

November 14, 2017 Source

Google may choose not to index URLs in sitemaps that are very similar to ones already being indexed and if they differ to the ones linked within the site (e.g. trailing slash/non-trailing slash).


Mixed Migrations May Cause Google to Index HTTP or HTTPS URLs

November 3, 2017 Source

Forgetting to update your sitemap files following a HTTPS migration could cause some pages to be indexed with the HTTP URL and some HTTPS.


Video Sitemaps Can Specify Countries Where Content is Available

October 31, 2017 Source

With a video sitemap you can define which countries your content is available, which is used for video search results.


Related Topics

Crawling Indexing Crawl Budget Crawl Errors Crawl Rate Disallow Last Modified Nofollow Noindex RSS Canonicalization Fetch and Render