Getting Page Performance Statistics in DeepCrawl

Alec Bertram
Alec Bertram

On 18th July 2018 • 3 min read

Our rendering engine is built to be as flexible as you need it to be. To help enable you to do whatever custom analysis you'd like to do on your website, it allows for custom Javascript files and snippets to be executed on every page we render.

 

Performance Statistics

Chrome, our rendering engine, gives us access to a whole host of page rendering performance statistics via Javascript.

Some of these are:
 
Performance timings

Read more about the performance timing numbers
 
Paint Timings

Read more about the paint timing numbers
 
Resource Statistics

 

Capturing these statistics

While we plan to include these metrics in DeepCrawl by default soon, a quick method to get these into your crawls now is using our custom functionality. Once you've decided which performance metrics you'd like to see in your rendered crawls, we need to capture these using a two-step process.

Grab the metrics using a custom Javascript injection
Add the following code snippet to the "Custom Script Injection" setting of your project:

var perfTimings=performance.timing.toJSON()
Object.keys(perfTimings).forEach((i)=>{
  if(perfTimings[i]>0){
      var newSpan = document.createElement("span");
      newSpan.id = "customextraction-perftimings-"+i
      newSpan.innerText = ((perfTimings[i]-perfTimings.navigationStart)/1000).toFixed(2)
      document.body.appendChild(newSpan)
  }
})
performance.getEntriesByType('paint').forEach(entry => {
          var newSpan = document.createElement("span");
      newSpan.id = "customextraction-perftimings-"+entry.name
      newSpan.innerText = (entry.startTime/1000).toFixed(2)
      document.body.appendChild(newSpan)
})
var deepCrawlResourceMetrics = {count:0,totalTransferBytes:0,totalDecodedBytes:0}
performance.getEntriesByType('resource').forEach(entry => {
    deepCrawlResourceMetrics.count++
    deepCrawlResourceMetrics.totalTransferBytes+=entry.transferSize
    deepCrawlResourceMetrics.totalDecodedBytes+=entry.decodedBodySize
})
Object.keys(deepCrawlResourceMetrics).forEach((i)=>{
    var newSpan = document.createElement("span");
    newSpan.id = "customextraction-resources-"+i
    newSpan.innerText = deepCrawlResourceMetrics[i]
    document.body.appendChild(newSpan)
})

Capturing these statistics with a custom extraction
The above step will dump the performance statistics into the DOM of your pages as we crawl them. The next step is to extract these into custom extractions.

Depending on the metrics that you'd like to save, use the following custom extractions:

Read more about the performance timing numbers
 
Paint Timings

Read more about the paint timing numbers
 
Resource Statistics

 

Using this data

Once your crawl has run, these metrics should be available in both the "Custom Extraction" reports, and the "All Pages" export.

The performance metrics are in seconds from the beginning of the render.

 

FAQs

 

Why are these metrics different to the ones in Google's CrUX/PageSpeed Insights/Lighthouse/etc?

Performance metrics are a way of breaking down a page's rendering lifecycle. These metrics tend to fluctuate as several factors can delay or speed up a page's rendering (such as network load, server performance, or arbitrary factors like which ads happened to load). DeepCrawl makes measurements while a page is rendering and saves these to be analysed later. As we render all pages from similar machines on the same network, this can provide a good comparative benchmark and gives insight into the pages which perform the best or worst across your website.

However, because these metrics are so specific to the hardware and network that the page was rendered on, you may find that there are discrepancies when retrieving the same data from other datasets such as Google's CrUX report, or Lighthouse run on your own computer. Google's CrUX report is based on renders across hundreds of user's machines; while this dataset provides a good overview of the average performance of a page, but the data can not necessarily be used to compare the performance of different pages (as different samples of users will be used for each page's metrics). A similar discrepancy will happen if you run Lighthouse on your own computer - the performance metrics will vary with the performance of your hardware, the other apps and webpages that you have active, and the types of browser plugins that you're using.

Author

Alec Bertram
Alec Bertram

Alec is the Head of Product at DeepCrawl. He has 10 years' of experience in SEO, and works to make sure we're building the most valuable things we can for our users.

Get the knowledge and inspiration you need to build a profitable business - straight to your inbox.

Subscribe today