DeepCrawl is on a mission

To provide enterprise-grade insights which go beyond SEO.

We believe that on-demand analysis of websites is valuable for more than just SEO optimisation. Over the next few years, DeepCrawl will expand to offer easy-to-configure, fast crawling of any website (yours as well as your competitors) and perform deep, complex analysis of these websites structure and content, to offer important and actionable insights not just for SEO efforts, but for all initiatives within Marketing teams, as well as for other teams. We will enter and disrupt these new markets in the same way that DeepCrawl already disrupted the SEO optimisation market.

We listen to our inner geek and take pride in good engineering. Over the last year, we completely rewrote the whole DeepCrawl platform and product, and are pretty happy with how it works now.

The more complex part of the platform is the crawling and analysis pipeline. Written in modular Ruby and using RSpec for tests, its architecture is based on dedicated services fulfilling different tasks.
These processes all run on AWS and we use a range of cloud services and storage solutions including S3, EC2, MongoDB, Postgres, Redis.
The user interfaces are really neat and light-weight, and written in Angular 1.x and Angular 2 (TypeScript), united tested with jasmine.js and served with express.js.

We have continuous integration in place (using Travis). Automated end-to-end tests run continuously. We deploy to production several times a week, and have good monitoring and alerting (Airbrake) in place. We use Puppet to simplify configuration and deployment.
We manage our priorities using a light version of Scrum. This allows us to make quick improvements to the platform and product without meetings getting in the way.