Segmentation in Deepcrawl

Helping our customers improve their SEO and earn impactful gains in their search results is at the heart of our business at Deepcrawl — and we’re constantly working on creating even faster, easier ways for our customers to execute their SEO strategies at scale and improve their websites’ performance. To that end, we have been making major improvements in our new application, with segmentation as a foundational element of the technology stack.
 

What is segmentation?

Segmentation allows you to easily conduct analyses on different page types across your website and efficiently identify SEO issues and opportunities.

With segmentation you can:

• Create custom segments based on Deepcrawl’s 250+ metrics
• Identify SEO issues quickly by only focusing on priority areas of your business
• Discover and monitor SEO trends in priority areas of your website.
• Export segment data into Google Data Studio using the Deepcrawl Connector to create customized dashboards

For example, you can create segments for different product categories on your website or for different country/language versions.

Getting started

Access to the new application is only available for enterprise clients. To find out more, please speak to one of our account executives.

There are 3 key steps to using segmentation:

1. Create your segments
2. Run a crawl
3. View the reports

How do I access segmentation in Deepcrawl?

You can access segmentation from the project dashboard or any of your Deepcrawl reports.

To switch to the updated version of the Deepcrawl app (and access our segmentation features), look at the example below. Within the report header, you will see a “New App” link. Click on it and you will be switched to our new app.

When you switch to the new Deepcrawl app, you will see our new report template and design. The first time you navigate here, the charts will be empty – to populate them you will need to create your segments.

How do I create segments?

Once you’re in the new app, you’ll simply need to select “new segment” from the drop-down menu in order to create and configure the desired segments for your project.

Creating a segment:

1. A segment must have:
   1. A name 
   2. At least 1 rule defined
2. You can create segments with most of the filters we have on our reports
3. If you choose regex validation, please see our hints and tips section
4. Once name and rules have been filled out, click “Create Segment”
5. The new segment will be added to the segment manager

Crawling & segmentation

Once you are happy with the segments you have created, then you must run a new crawl. To do this, just navigate back to step 4 in Project Settings and run a new crawl from there:

Notes on generating a segment:

1. A new crawl must be run for a segment to be generated.

2. You can also now preview a segment before creating it:

3. A segment begins to generate after a new crawl has finished.

4. You can create segments during the “crawling” state and they will also generate.

5. If you create a segment during the “finalizing” state, however, you will need to run a new crawl for a segment to generate.

(An example of a crawl during the “finalizing” stage is below — create your segments before this stage, or before running a crawl.)

6. You can create segment rules not only by URLs ,but also by any other metric available in Deepcrawl reports:

7. You can also use “AND” / “OR” statements in your segments:

8. Now both “Issues” and “Changes” reports can also be segmented in Deepcrawl:

Viewing reports

After you have run your crawl and your segments have been successfully generated, you can navigate back to “Reports” to view. You will initially be taken to reports that display the following data:

• Top segments in a report
• Trend graph (max 30 crawls).

You can then select a specific segment by clicking the “view segment” radial and selecting which segment you wish to analyze. Alternatively, you can click on any data bar within the charts.

When a segment is applied, the data in the report will be updated based on the rules in the segment.

The applied segment is also persistent across the new core app, which means that you can navigate to another report and the segment is still applied.

If the segment has been generated in past crawls then it will even persist when you switch to an unarchived historic crawl:

You can also apply local filters on top of your segments:

Exporting segment data

When a segment is applied, the data can be exported into a CSV file. Simply go to the “Export data” button and select “Generate CSV [Segment]”.

The CSV file will generate and then be downloaded to your browser.

As well as downloading the segment report, you can also download the segment report with filters added:

If you want to delete any downloaded files,  simply click on the bin icon to remove them from the downloads dropdown.

Graphs

Top Segments graph:

The Top Segments graph displays the top 5 segments that have the most URLs in a report. It can be used to quickly highlight problem areas of a website in any report.

The data points are clickable so you can simply click on any segment in the graph and it will take you to that segment in the report (which will persist when you move to another report in the new core app).

Segment Report Graph

The segment report chart compares the currently selected segment URLs against all other URLs in this report. It allows customers to see how big the URLs are within the context of the whole report.

If you have only created a handful of segments and these segments are quite small in the context of the rest of the pages in a report, then it indicates that more segments need to be created.

Trend Chart

The trend chart maps out the data trends over the last 30 crawls for any segment report. It can help you track improvements or issues for particular areas of the website.

For example, the number of pages over the medium fetch time being crawled spiked in the last crawl in the /2020/ area of a large publisher website.

Additional Hints and Tips for Segmentation in Deepcrawl

Regex validation

If you choose “Matches (Regex)” and “Doesn’t Match (Regex)” rules, please enter Lucene valid regular expressions. Use Rubular to help write valid regular expressions. Forward slashes do not need to be escaped.

Managing and copying segments

1. You can delete a segment, a tooltip will always ask if you want to delete to ensure you don’t accidentally delete a segment
2. You can view rules for each segment created
3. You can see when a segment was generated
4. You can bulk copy a group of segments to another project in your account.

Author

Helen Cherry