Modifying URLs And Stripping Parameters

Tristan Pirouz
Tristan Pirouz

On 11th October 2016 • 2 min read

You can make modifications to the URLs, as they are being crawled, using the ‘Remove URL Parameters’ and ‘URL Rewriting’ features in Advanced Settings, in step 4 of the crawl setup.

These features are useful to undertake tasks such as removing URL components that are complicating analysis of your website or to rewrite URLs to an external website, such as lookup service e.g. retrieving information from an API for a set of your page URLs.


Stripping URL Parameters

If you simply want to strip out parameters, you can list them on separate lines in the Remove Parameters option in Advanced Settings. e.g. Add ‘param1’ to strip all parameters such as param1=1, param1=2, etc.


URL Rewriting

The URL rewriting function allows you to use regular expressions to modify your page URLs in more complex ways.

The URL is matched by the regular expression in the ‘Match From’ column, and replaced with what you set in the ‘Match To’ column.

If you use parentheses in the Match From setting, you in conjunction with variables $1, $2, etc. Deepcrawl will insert whatever text matches the corresponding parenthetical group.


Append URL (must be in this order):

Match From1: (.+?.+)
Match To1: $1&

Match From2: (^[^?]+$)
Match To2: $1?


Replace a domain:

Match From: (^https?://)(www.|)
Match To: $


Change the name of a parameter:

Match From: parameter_x=([^&$]+)
Match To: parameter_y=$1


Change the case of a parameter:

Match From: parameter_x=([^&$]+)
Match To: parameter_x=$1
Case Options: uppercase


Change HTTPS to HTTP:

Match From: ^https:
Match To: http:


Force Trailing Slash:

Match From: ([^/])$
Match To: $1/

Writing regular expressions can get tricky, so contact us at if you need any help with these features.


Tristan Pirouz
Tristan Pirouz

Tristan is a Former Head of Marketing at Deepcrawl.

Choose a better way to grow

With tools that will help you realize your website’s true potential, and support to help you get there, growing your enterprise business online has never been so simple.

Book a Demo