Sitemaps

A sitemap is a list of all of the live URLs which exist on a site and is used to inform search engine crawlers of the most important pages and therefore which ones should be crawled and indexed. There are several things to consider when creating sitemaps, as well as understanding how search engines view them. We cover a range of these topics within our Hangout Notes, along with best practice recommendations and advice from Google.

Use X-Robots-Tag HTTP Header to Noindex Indexed Sitemap Files

October 19, 2018 Source

If sitemap files are indexed for normal search queries, then you can use the X-Robots-Tag HTTP header to noindex all pages ending in .xml or .gz.


Compressing Sitemaps Saves Bandwidth But Doesn’t Reduce Processing Time

October 19, 2018 Source

Compressing sitemap files using Gzip can save bandwidth but doesn’t impact the speed that Googlebot processes these files.


Check URLs Match Exactly When GSC Reports a URL is Not in a Sitemap

October 19, 2018 Source

If the URL Inspection tool says a page is indexed but not submitted in the XML sitemap, John recommends checking that the exact URL seen in Search Console is present in the sitemap. For example, you should check that there are no differences with trailing slashes or the case used.


Update Last Modified Date in Sitemap & Use Validate Fix in GSC to Get Pages Crawled Sooner

October 5, 2018 Source

If technical issues cause pages to show incorrectly (e.g. serving a blank page), you can get Googlebot to crawl these sooner by submitting sitemap files with the last modification date set to when the affected pages were restored. You can also click ‘validate fix’ on pages with errors in Search Console to get Googlebot to recrawl these pages faster.


KML Sitemaps Not Supported by Google

September 18, 2018 Source

KML sitemaps files are not supported by Google and haven’t been for some years.


Empty Sitemap Files Should Be Removed

August 24, 2018 Source

Empty sitemap files with no URLs are not problematic for Google but John recommends removing them if they are not needed.


You Don’t Need to Submit Separate Mobile URLs in Sitemaps

August 21, 2018 Source

URLs for separate mobile sites don’t need to be submitted in their own sitemap. If the rel alternate tags are set up correctly then Google will be able to find these pages through their connection to their desktop equivalents.


Google Have Removed Their Public URL Submission Feature

July 27, 2018 Source

Google’s public URL submission tool has been removed but URLs can still be submitted through Search Console and sitemaps.


Image Sitemaps Associate Images With Pages But Don’t Provide Context For Images

July 27, 2018 Source

Image sitemaps are useful for Google as they show what images are associated with what landing pages, but they don’t provide context about the images themselves like alt text and image titles do.


Related Topics

Crawling Indexing Crawl Budget Crawl Errors Crawl Rate Disallow Last Modified Nofollow Noindex RSS Canonicalization Fetch and Render