DeepCrawl allows users to crawl XML Sitemap URLs and any URLs found in a Sitemap file. 
To add XML Sitemaps to a new crawl project follow these instructions:

Start a new crawl project

Create a new crawl project in DeepCrawl. Create new project DeepCrawl Enter the primary domain where the XML Sitemap files are located and name the project (enable JavaScript rendering if you want to use the Page Rendering Service). Crawl setup DeepCrawl

Source Settings

In the Sources settings select the Sitemaps option. By default DeepCrawl discovers and crawls XML Sitemaps using the /robots.txt file. Source settings DeepCrawl

How to add XML Sitemaps to DeepCrawl

When adding Sitemaps to a crawl project, there are a number of options which helps DeepCrawl discover Sitemap URLs:
  1. XML Sitemaps referenced in the /robots.txt file (this is done by default).
  2. Manually copy and paste XML Sitemap URLs into DeepCrawl.
  3. Manually upload .xml or .txt files into DeepCrawl.
Add sitemaps DeepCrawl

Discover XML Sitemaps found in /robots.txt file

By default, DeepCrawl will discover any XML Sitemap URL in a /robots.txt file when selecting the Sitemaps data source. Select sitemaps DeepCrawl This means that any Sitemap URLs found in the /robots.txt will be automatically pulled into the list of Sitemaps DeepCrawl will crawl.
For example the following /robots.txt file: User-agent: * Sitemap:
DeepCrawl would discover the Sitemap Index and crawl all the XML Sitemaps contains within that file. Any Sitemaps not included within that Sitemap Index would not be discovered and crawled by DeepCrawl in this example. If you do not want DeepCrawl to discover XML Sitemaps in the robots.txt file in future crawls then unselect it in the Sources settings. Sitemaps robots.txt DeepCrawl Remember to deselect the Sitemaps found in the /robots.txt, as they are not automatically deselected when deselecting the robots.txt option.

Add XML Sitemap URLs Manually

If XML Sitemaps cannot be found in the /robots.txt, then they can be added into DeepCrawl manually. To add Sitemaps manually to DeepCrawl:
  1. Create a list of all your XML Sitemaps.
  2. Copy all the XML Sitemap URLs and paste them into the into the input box.
  3. Click Add Sitemaps.
Manually add sitemaps DeepCrawl When the new Sitemaps are added manually, they will be added to the list of XML Sitemaps which DeepCrawl will discover and crawl. Select sitemaps DeepCrawl Once a list has been generated, DeepCrawl also gives you the option to select or deselect the files from being crawled in the project. Select sitemaps DeepCrawl This is useful if you don’t want to crawl every single Sitemap uploaded.

Upload Sitemaps

DeepCrawl gives the option to manually upload .xml and .txt files, which can be used as XML Sitemaps. This option is very useful if web development and technical SEO teams who wish to test Sitemaps before they go live. To upload a .xml or .txt file:
  1. Create a list of URLs you want to include in the .xml or .txt Sitemap.
  2. Create an XML Sitemap file or .txt file and include the URLs you want crawling by search engines (follow the Sitemap best practices guidelines).
  3. Upload the Sitemap file to DeepCrawl (no more than 100MB per file upload and DeepCrawl accepts gzipped compressed files).
Once a file is uploaded DeepCrawl will display the Sitemap file names and file extensions in a list. Sitemap upload DeepCrawl DeepCrawl allows you to edit this list of uploaded files, from the Source settings you can:
  1. Download the file to make sure the contents are correct.
  2. Delete the file in case the wrong file was uploaded.
  3. Deselect the file from being crawled in the project.

Configure crawl limits and advanced settings

Once the XML Sitemaps have been added to the crawl project, then save and continue and configure the crawl limits and advanced settings. Hit save, start the crawl and wait for the project to finalise to see XML Sitemap data included in the crawl.


Sitemaps can be added to new projects and can be uploaded in multiple different ways. When adding XML Sitemaps to a project:
  1. Start a new crawl project.
  2. Select Sitemaps in the Sources settings in a project, as well as other data sources.
  3. Choose which option to add an XML Sitemap (manual, file upload or through robots.txt).
  4. Save the Sources settings and configure the crawl limits and advance settings.

Recommended guides