DeepCrawl is the most advanced crawler on the market and is an essential part of any search marketer’s toolstack. With all of our advanced features, configurations and reporting, we know that you might need a helping hand getting started with our crawler.
To help ease you into crawling with DeepCrawl, we thought it would be helpful to give you some guidance on the various aspects that you will likely want to get to grips with.
So, what can we help you with?
- 1. How Can I Crawl a Website?
- 2. How Can I Deal With Crawl Errors or Credit Issues?
- 3. How Can I Use Crawl Limits to Reduce Credit Usage?
- 4. How Can I Change the Size and Depth of my Crawls?
- 5. How Can I Restrict a Crawl to Certain Pages?
- 6. How Can I Enrich My Crawls Using Regex?
- 7. How Can I Crawl a Test Site?
- 8. How Can I Crawl a Separate Mobile Site?
1. How Can I Crawl a Website?
We’ve written a thorough guide on how to crawl a website with DeepCrawl which is a great place to start if you’re new to setting up crawls. In the rest of this walkthrough you’ll learn how to perform a technical site audit using DeepCrawl to understand your website’s technical health and performance in organic search.
If you’re interested in crawling multiple domains or subdomains then we have a separate guide which shows you how to configure this in DeepCrawl.
2. How Can I Deal With Crawl Errors or Credit Issues?
Are your crawls generating errors, or are you having issues getting a crawl to reach the required level of URLs? Here’s a guide on troubleshooting failed crawls.
3. How Can I Use Crawl Limits to Reduce Credit Usage?
Don’t forget to use crawl limits so you don’t waste crawl budget, especially on large sites or where your site has resources of low value. If you’re having issues with URL parameters or want to rewrite URLs on the fly for lookup services using APIs, we’ve got a guide for that too.
4. How Can I Change the Size and Depth of my Crawls?
Whether your site has several million URLs or only a few thousand, crawling the entirety of a website isn’t always the best plan of attack. Knowing when and how to restrict your crawls to specific parts of a website is a fine art that will help you use DeepCrawl more effectively, as well as using up fewer credits.
Our guide on limiting the size and depth of a crawl is a great introduction on how to crawl more efficiently with DeepCrawl.
5. How Can I Restrict a Crawl to Certain Pages?
There are some cases where you are only going to be interested in focusing your crawls on a specific area of a site. You might only want to crawl one country version of a large international site, for example.
If you need to learn about how to restrict crawls using URL filtering, page groupings or regex, we’ve got your back.
6. How Can I Enrich My Crawls Using Regex?
Regex can be a love/hate thing… Once you truly understand it you will love it, especially with the extra power it gives you within DeepCrawl. From identifying thin content, to finding unique data about your products and copy, this post will detail how you can use custom extractions to get more out of your crawls.
7. How Can I Crawl a Test Site?
From a full migration to simple updates, code regression and plain bugs impacting SEO performance are still very common. By following our guide to crawling your live and test sites you can analyze and avoid any potential setbacks.
8. How Can I Crawl a Separate Mobile Site?
Separate mobile sites can be slightly more complex to crawl compared to other mobile configurations. If you have user-agent redirects that automatically take you to the correct version of the site, you’ll need to specify a separate mobile user agent in your crawl setup.
Our release post covers the ins and outs of setting up a secondary user agent so you can the crawl your desktop and mobile sites within the same crawl We’ve even got a product guide that delves more generally into crawling a mobile site.