Deepcrawl is now Lumar. Read more.
DeepcrawlはLumarになりました。 詳細はこちら

How to Add XML Sitemaps to Existing Crawl Projects

Lumar allows users to crawl XML Sitemap URLs and any URLs found in a Sitemap file.
To add XML Sitemaps to an existing crawl project follow these instructions.
 

Update an Existing Crawl Project

Navigate to an existing crawl project in Lumar and then click on the cog icon in the top right, then click on the Edit button.

This will take you to the first step in the project’s settings. Navigate to step 2 either by clicking save and continue or clicking the 2 icons.


 

Source Settings

In the Sources settings select the Sitemaps option. By default, Lumar discovers and crawls XML Sitemaps using the /robots.txt file.


 

How to Add XML Sitemaps to Lumar

When adding Sitemaps to a crawl project, there are a number of options that help Lumar discover Sitemap URLs:

  1. XML Sitemaps referenced in the /robots.txt file (this is done by default).
  2. Manually copy and paste XML Sitemap URLs into Lumar.
  3. Manually upload .xml or .txt files into Lumar.


 

Discover XML Sitemaps Found in Robots.txt File

By default, Lumar will discover any XML Sitemap URL in a /robots.txt file when selecting the Sitemaps data source.

This means that any Sitemap URLs found in the /robots.txt will be automatically pulled into the list of Sitemaps Lumar will crawl.

For example the following /robots.txt file:
User-agent: *
Sitemap: https://www.example.com/sitemap_index.xml

Lumar would discover the Sitemap Index and crawl all the XML Sitemaps contains within that file. Any Sitemaps not included within that Sitemap Index would not be discovered and crawled by Lumar in this example.

If you do not want Lumar to discover XML Sitemaps in the robots.txt file in future crawls then unselect it in the Sources settings.

Remember to deselect the Sitemaps found in the /robots.txt, as they are not automatically deselected when deselecting the robots.txt option.
 

Add XML Sitemaps Manually

If XML Sitemaps cannot be found in the /robots.txt, then they can be added into Lumar manually.
To add Sitemaps manually to Lumar:

  1. Create a list of all your XML Sitemaps.
  2. Copy all the XML Sitemap URLs and paste them into the input box.
  3. Click Add Sitemaps.

When the new Sitemaps are added manually, they will be added to the list of XML Sitemaps which Lumar will discover and crawl.

Once a list has been generated, Lumar also gives you the option to select or deselect the files from being crawled in the project.

This is useful if you don’t want to crawl every single Sitemap uploaded.
 

Upload Sitemaps

Lumar gives the option to manually upload .xml and .txt files, which can be used as XML Sitemaps.

This option is very useful if web development and technical SEO teams who wish to test Sitemaps before they go live.

To upload a .xml or .txt file:

  1. Create a list of URLs you want to include in the .xml or .txt Sitemap.
  2. Create an XML Sitemap file or .txt file and include the URLs you want crawled by search engines (follow the Sitemap best practices guidelines).
  3. Upload the Sitemap file to Lumar (no more than 100MB per file upload and Lumar accepts gzipped compressed files).

Once a file is uploaded Lumar will display the Sitemap file names and file extensions in a list.

Lumar allows you to edit this list of uploaded files, from the Source settings you can:

  1. Download the file to make sure the contents are correct.
  2. Delete the file in case the wrong file was uploaded.
  3. Deselect the file from being crawled in the project.

 

Run or Schedule an Existing Crawl Project

Once the XML Sitemap data has been added to the crawl project click save and continue.

The next time the crawl project is run Lumar will use the XML Sitemap URL data, and Sitemap reports will be populated.


 

Summary

Sitemaps can be added to existing projects and can be uploaded in multiple different ways. When adding XML Sitemaps to a project:

  1. Navigate to the advanced settings in a project.
  2. Select Sitemaps in the Sources settings in a project, as well as other data sources.
  3. Choose which option to add an XML Sitemap (manual, file upload or through robots.txt).
  4. Save the Sources settings and configure the crawl limits and advance settings.

 

Recommended Guides

Avatar image for Adam Gent
Adam Gent

Product Manager & SEO Professional

Search Engine Optimization (SEO) professional with over 8 years’ experience in the search marketing industry. I have worked with a range of client campaigns over the years, from small and medium-sized enterprises to FTSE 100 global high-street brands.

Newsletter

Get the best digital marketing & SEO insights, straight to your inbox