The Author
Rachel Costello

Rachel Costello is a Technical SEO Executive at DeepCrawl. You'll most often find her writing and speaking about all things SEO.

Read more from Rachel Costello

Ask the Expert: John Doherty Answers Your SEO Monitoring Questions

For our most recent webinar, John Doherty joined us to give a presentation on SEO monitoring. On top of this, he also answered some of the questions that were submitted to him by the attendees during the session. John only managed to answer some because there were just too many for him to be able to answer live!

Luckily, John kindly took the time to write out responses to all of the questions he didn’t get round to answering during the webinar. We’ve included them all in this post, so read on to learn John’s expert advice on SEO monitoring best practice and insights into his own methods.

Which rank tracking tool is the most accurate?


A rank tracking tool should show you desktop and mobile rankings as well as the different SERP features like featured snippets etc. The two I use the most for this are STAT and SEMrush.

Can you explain the concept of keyword buckets?


On my site I have different page types like category, resource and blog pages. Ecommerce sites will also have a lot of category and subcategory types for grouping pages such as black trousers and blue trousers. With a keyword bucket, you would group these two subcategories together into a ‘trousers’ bucket where you would track related keywords at a group level.

How do you suggest monitoring changes at scale (e.g. 100k pages or more)?


This depends on what the changes are and if you need a full crawl each time, or what specifically you are monitoring (how well pages are indexed? Rankings? Technical issues?) Most tools can scale to that count of pages relatively easily.

What percentage of your revenue (maximum) would you spend on tools?


This is a good question that is unfortunately impossible to answer because percentages change depending on your revenue! You can have a $1M/yr revenue company that is spending the same amount of money on tools as a $100k/yr revenue company, and percentages are going to be wildly different even though they are spending the same amount.

How I think about this is that I am willing to invest in new tools after I’ve done my research and they fill a gap in my toolset that needs to be filled. However, I also go back every month and review my spending across all categories including tools, and if I have not used a tool for 3 months then I eliminate that spend.

What do you recommend for assigning a monetary value to conversions for industries outside of ecommerce?


CEOs understand dollars, and when we are working on SEO we need to do our best to point the way back to what the efforts we are doing means for the company.

It is MUCH easier to assign dollar values to a specific channel like SEO when you are dealing with something like ecommerce as opposed to a business type like healthcare or even any service-based business where a sales team is involved.

The truth is that tracking will be a lot harder to assign dollar values to this, but it is possible. If you have a sales team then they probably have a CRM, and most CRMs will also let you assign a source automatically such as Direct, Referral, Organic, and the like. You can also run reports in most CRMs to help you understand how well leads from different channels convert.

I would also say that you can report on micro-conversions, which are basically events along the way that move towards that ultimate sale/conversion that becomes realized revenue. These are things like overall leads (however you define that), MQLs (marketing qualified leads), and then tracking them on down the funnel to SQLs (Sales Qualified Leads) and ultimately customers.

Any suggestions for monitoring many small sites (20+) at scale?


Most tools can do that pretty easily depending on your subscription level. If you’re an agency and have 20 or more clients, then you’ll need a subscription to a tool like DeepCrawl or Moz that allows you to have that many campaigns/projects. Amortize this over all of your clients (if it costs $500/month for 20 clients, that’s $25/client) and pass this cost along.

I also recommend that your clients have their own subscription to a tool that they pay for so that when/if they leave your services and either do it themselves or move on to someone else, they have that data that they own.

What is your best recommendation for someone starting out with minimal knowledge of SEO? What’s the best resource for learning to fix crawl errors?


If you’re starting out with minimal knowledge of SEO then you need to educate yourself about how it works. Start with the Moz Beginner’s Guide to SEO, maybe enroll in DistilledU to learn a lot of the technical areas and things like keyword research, and read as much as you can on Moz, Search Engine Land, Search Engine Journal, and similar sites.

As far as the best source for teaching you how to fix crawl errors, that is a tough one to answer because the “right” fix for a technical error is going to depend on what platform you are on, what language your site/app is written in, and a host of other things. This is why different people specialize in specific platforms like Shopify or WordPress or SquareSpace so that they can learn the limits and benefits of them inside and out.

What I say is that if you are not technical, then you do not need to worry about how to code. Your role should be to diagnose what is sub-optimal technically, talk about why it should be fixed (e.g. how it’s a large problem keeping pages from being crawled and that you should fix it with the goal of more pages indexed and more traffic), and then work with a developer to scope it (difficulty, effort) and learn as you go around what is possible and what is not.

What are your thoughts on Spyfu and Ahrefs?


I use Ahrefs quite often and pay for a subscription there as well. It’s a great tool and I use it for specific things that other tools are harder to do. For example I love Ahrefs’ keyword research tools and insights into subdomains.

It has been a long time since I have used Spyfu. It just has never been a big part of my toolset. The tools I mentioned during my presentation should help you cut through the noise and select the tools you really need.

What are your thoughts on Google Data Studio?

Google Data Studio is a pretty new tool that most people haven’t leveraged yet, and I have not either. I think it is a great dashboarding option, though there are also others that can give you the same data.

The tool is all in how you use it and if it’s right for your specific situation.

Why not use Google Search Console to track rankings?

Google Search Console’s rankings are not nearly as accurate as other paid tools out on the market. GSC’s rankings are averages, but they’re not updating daily (as far as I know) and you also don’t get insight into paid ad spots, featured snippets, SERP features like “People Also Ask”, and the like.

Overall I would say GSC is a data point and I appreciate the data, but it is often not specific enough and I cannot drill further down to get the really good insights needed.

Are you using any tool to store Google Search Console data and create insights inside it?

I am not, but there are a few out there and I’ve seen some scripts around the internet that you can use to store the data in a Google sheet daily. This is probably something I should look into doing.

What about page speed – do you monitor it? If so, with which tool?

I do monitor it on my site for specific types of pages (e.g. homepage, resource guide, categories) and I personally do it using SanityCheck.io (I have a free account). I also know DeepCrawl has integrated this tracking now. This guide explains how you can use the tool to collect Chrome page speed metrics at scale.

To learn more about site speed and performance optimisation, make sure you take a look at our white paper on the topic.

What’s the best and smartest way to track competitor’s backlinks?

You can use pretty much any of the tools I mentioned to do this – Moz, SEMrush, Ahrefs all let you do it pretty easily. You can also set up Google Alerts or alerts through your tool of choice for their brand term so you can see where they are being mentioned which can inform your tracking of their promotional activities that help SEO.

How would you present an SEO report to your manager if you are in-house? What values would you want to show?

This really depends on which metrics your manager cares about as well as which metrics matter to the business. I’ve had managers where I’ve reported just unique users. I’ve had other managers where we reported number of leads from organic. I also usually like to include something about the strategic plan in reporting as well, such as what metrics we are looking to affect positively in the future and what the plan is for getting there.

How would you keep yourself updated if traffic slowly fell e.g. 10% week on week?

I don’t do the -10% as an example because often traffic can vary like that when there is a national holiday or something similar. 20% is big enough that it probably merits a hard look and a heads up, but 10% usually doesn’t worry me unless it stays that low for a few days and it can’t be explained by something else. I also check Analytics manually every weekday, so it would be hard to miss a drop.

We have universities as clients, does that mean that nursing, MBAs and education would be my keyword buckets?

If the university is your client and you are tracking keywords across those verticals/areas of study, then yes those would be your keyword buckets. You could also go a level deeper and do Nursing and Masters as an example, so you can cut the data multiple ways to see what is moving.

When looking at your subscription crawl budget, what percentage should you dedicate to monitoring your competitors?

Your site is your main concern and your moneymaker. I would say that the majority of your crawl budget should go into your own site because you can directly affect that. I assume you are also looking at your competitors from your browser and in other tools you use, so you’ll see major shifts that they make.

So I would say crawl your own site weekly (or at least a portion of it to monitor it well), and then crawl your competitors on a monthly or bi-monthly basis. This should be enough to understand what is going on with them, but also lets you focus on your own site.

Join our next webinar on international SEO with Aleyda Solis

Aleyda Solis internationalisation webinar with DeepCrawl
If you enjoyed this webinar on SEO monitoring, then you’re in for a treat with our next one! International SEO expert, Aleyda Solis, will be sitting down with Jon Myers to explain everything you need to know about internationalisation and succeeding globally online. This is set to be an excellent session so make sure you reserve your place now.

Book your spot now