Deepcrawl is now Lumar. Read more.
DeepcrawlはLumarになりました。 詳細はこちら

How to Measure Indexed Pages More Accurately

SEO and Digital Marketing Best Practices

If the need arises to check how many of a site’s pages are indexed (ie. those URLs that are returned in search results) many SEOs’ first reaction is to fall back on the site: query, but in fact this query returns all URLs that Google is aware of, including canonicalized pages, so is probably inaccurate.

site query

With this in mind, let’s go through some more accurate ways to find indexed pages, and then a method for combining them to find a more accurate final figure.

accuracy: dart board
 

Search Console

 

Index Status report (Google Index > Index Status)

This high-level report will show you the number of pages in the index, which can be compared against your own counts, although it can include duplicates that might be filtered out of ‘real-world’ search results.

google search console indexed pages report
 

Sitemap indexation count (Crawl > Sitemaps)

Will show you an indexation count for your Sitemaps. If the number of submitted pages matches the number of indexed pages, you know that 100% of the URLs are indexed. If you have Sitemaps with fewer files in each, you are more likely to be able to make use of this information.

google search console sitemaps report
 

Unique URL Sitemap indexation count

Google’s John Mueller recommends creating a Sitemap file containing all of your unique URLs, then submitting this in Search Console. This will give you more accurate information on the number of unique indexed URLs.
 

Pages with SERP impressions (Search Traffic > Search Analytics)

Anything getting a search impression must be indexed.
 

Google Analytics

Every page that has received a visit from Google in a recent period is almost certainly still indexed. Although not 100% reliable, it’s very accurate, depending on the timeframe you’re measuring.
 

Ranking checking

Any URL that’s ranking in search results must be indexed: use your preferred ranking checker to download more indexed URLs that might not be picked up in other ways.
 

Info: query

This is the more accurate way to check whether a page is indexed, however it can only be used to check one URL at a time.

info query
 

Combinations

 

1. Download and combine all the URLs from these sources in Excel, then de-dupe:

google analytics organic landing pages report
 

2. Crawl the URLs as a List Crawl to get the Unique Pages count:

You can either paste up to 2,000 URLs as a list, or upload a file.

deepcrawl list crawl
 

3. You can then compare this number of unique URLs to the number reported in the Indexed Pages count in Search Console.

 

Indexation: further reading

Avatar image for Tristan Pirouz
Tristan Pirouz

Marketing Strategist

Tristan is an SEO enthusiast, strategist, and the former Head of Marketing at Lumar.

Newsletter

Get the best digital marketing & SEO insights, straight to your inbox