In the following article I will take a look at the relevance of DeepCrawl for agencies.

I must admit: I like DeepCrawl a lot. For me as an agency consultant, DeepCrawl is an efficient tool that provides exactly the information that I need. It simplifies audits and, as well as the operational tasks for various projects due to its features and the data it provides. Here we go:

The dashboard: important information at a glance

The DeepCrawl dashboard is its centrepiece and differentiates itself from other good tools.

Look at these examples:

By giving you an instant overview on the main issues at a glance, you can assess the condition of a domain very quickly. When I see data presented like this, I have many ideas about where to go next and precisely this is what a dashboard should do.

Additionally, if you are working closely with a custumer and both of you open the dashboard on your own screen, you often do not need to explain what “might” be wrong, as the indicators are clear and intuitive.

Sharing Reports: a small, but efficient feature

DeepCrawl allows users to share their results with others by providing links to their data view pages. Instead of exporting big data files and sending them via email, you can share all necessary information by forwarding the link and the recipient can retrieve the data as well as export it – they don’t even need an account. Especially for a consultant like me or even for an inhouse-SEO, it is a function that saves a lot of time, which is what a tool also needs to do: optimize your workflow.

No verification needed: start as soon as you like

Even if it sounds simple: having the option to crawl all the websites I want without any restrictions is pretty important to me. With DeepCrawl, this is possible as it waives the verification process. Although I can understand that some tool providers request a domain verification at first, for me as a consultant it would only cause problems. In many companies it takes a lot of time to receive the written verification: tickets and mails have to be written to the IT department to get approval, including a code, then it must be scheduled in the priority process, then the IT-guy is ill – it’s all happened.

To ensure a good workflow, I need the data as soon as possible and so a quick-starting process is essential to me.

Page Grouping: save credits and crawl broader

If you work with websites with several million indexed pages, it is often not efficient (and also a more expensive use of credits) to crawl them in their entirety when a sample would be enough to get a feel for what’s happening on the site. This also allows you to crawl several folders instead of just a big one. The good thing here: DeepCrawl allows you to set limits on specific sub-folders and save credits. So if you have a domain with some really big sub-folders, limit the number of URLs the crawler is allowed to crawl within this specific sub-folder. In this way you can crawl a wider variety of sub-folders from the same domain.

Crawling live vs. test environment: you don’t need to wait until it is live

One really cool feature, which I like a lot, is comparing the live status quo of a domain with the test environment. If you are facing a relaunch, an adaptation of the internal linking structure or if you just want to add a new website segment, this feature helps you get an overview of the ways your domain and the crawling of your domain may change when you release. Here you can see if your changes would result in an increase in nonindexable pages, canonicalized pages, 404 linking errors, non 301 redirects and so on.

One thing you should definitely always pay attention to is the status of the web depth chart on the dashboard. Thanks to this chart you immediately see if a domain has an issue with URL-overhead for example, or if the structure of a domain is “broken” because the crawler found 100 levels on your domain. The following three charts will illustrate what I mean.

Here you see a domain with a significant overhead concerning URLs. The amount of unique URLs is lower than the number of non-indexable URLs. A structure like this cannot be efficient.

Here you see a domain with at least 100 levels. This means that something in the structure is broken as the crawler finds a high amount of displaced sites.

This diagram shows another site with a tremendous overhead concerning URLs. The crawler needs to crawl several hundred thousand URLs more than might be needed. This is also known as crawl bloat and search engine bots may not be discovering key pages on your site as it has spent its time crawling unnecessary URLs.

Another great feature that DeepCrawl has to offer is trend monitoring. You are able to see the changes of certain issues over time, such as unique pages or duplicates in a diagram – and believe me, nothing is more convincing than graphs and data when it comes to customers and board meetings. If you can reveal that a specific issue has improved over time or from one release to another, you win.

Work with different projects within a domain: It’s smart crawling

If you are working continuously on a specific domain, it often makes sense to schedule sub-crawls. For example, if you have a domain, which is available in several languages or has different product types, it is wise to schedule a crawl project for each language/country/product type.

The advantage of this is that some issues are site-wide and if you recognize them in one country, they will often exist in other countries too. Additionally, if the countries are treated separately it saves you credits using different projects when you need to check country-related changes. But apart from that, what you could also do is to set up projects for a specific type of URL, e.g. crawl URLs with a bounce rate of 100% within the last 12 months. So think about which type of URL is suited for a special crawl.

Scheduled Crawls: there will be hectic times

Even if it is a “no-brainer” – do it! Scheduling crawls for long-term customers makes a lot of sense. During hectic times you will be thankful for the automation. With scheduled crawls you will never be in the situation of not having fresh data when it comes to a spontaneous call from a client. Additionally, as shown above, DeepCrawl allows you to create charts with the development over time, which is always helpful.

If you have a set of the most important URLs of a domain, it is always worth the time and effort to have a look at the “No links in” row. With this metric you can easily recognize problems regarding internal linking and prioritisation. If a URL with a very low DeepRank (an internal measurement of link authority) is one of your best-linked URLs – you might have to think about it.

Finally, whether a tool is helpful or not depends on your own situation. As previously mentioned, for me as a consultant there are some really crucial reasons why I favour DeepCrawl. However, the best thing you can do is to give it a try and see where this brilliant tool can support you in your daily work. Hope you had some nice takeaways!