Global leader in Web Crawl

DeepCrawl is the most comprehensive webcrawler in the world. An award-winning solution adopted by more than thousands of clients

DeepCrawl is on a mission

To provide enterprise-grade insights which go beyond SEO.

We believe that on-demand analysis of websites is valuable for more than just SEO optimisation. Over the next few years, DeepCrawl will expand to offer easy-to-configure, fast crawling of any website (yours as well as your competitors) and perform deep, complex analysis of these websites structure and content, to offer important and actionable insights not just for SEO efforts, but for all initiatives within Marketing teams, as well as for other teams. We will enter and disrupt these new markets in the same way that DeepCrawl already disrupted the SEO optimisation market.

Founded by London-based SEO gurus to solve their own business challenges

2010

Bootstrapped for 3 years using professional services

2011

Incubated by Sandpit, backed by HNWs

2013

Beaumont Capital investment

2014

New offices, more tech, sales, marketing

2015

DeepCrawl 2.0: smarter, faster, more scalable

2016

We listen to our inner geek and take pride in good engineering. Over the last year, we completely rewrote the whole DeepCrawl platform and product, and are pretty happy with how it works now.

The more complex part of the platform is the crawling and analysis pipeline. Written in modular Ruby and using RSpec for tests, its architecture is based on dedicated services fulfilling different tasks.
These processes all run on AWS and we use a range of cloud services and storage solutions including S3, EC2, MongoDB, Postgres, Redis.
The user interfaces are really neat and light-weight, and written in Angular 1.x and Angular 2 (TypeScript), united tested with jasmine.js and served with express.js.

We have continuous integration in place (using Travis). Automated end-to-end tests run continuously. We deploy to production several times a week, and have good monitoring and alerting (Airbrake) in place. We use Puppet to simplify configuration and deployment.
We manage our priorities using a light version of Scrum. This allows us to make quick improvements to the platform and product without meetings getting in the way.

Join us in solving some of our biggest problems yet


            

Responsibilities


            

Requirements


            Apply