Deep Crawl v Screaming Frog

As the original post was over 2 years old, I thought it was time to update this article as so much has changed in the meantime. You will now see the latest info on both the  products

Which is best? Deep Crawl or Screaming Frog?

I’ve used both pieces of software and I love them both, although there are also other options out there.

What is screaming frog SEO Spider

Screaming frog is a desktop application you can download to crawl your website to find technical issues. Screaming frog works by sending a bot to your website to crawl it and reports back certain technical reports. It’s how you use these reports which means you can use the screaming frog tool to boost your SEO efforts.

So whats the differences between Screaming Frog and DeepCrawl.

Screaming Frog

There is a free version of Screaming Frog, but at £149 per year, the paid version is worth the investment.

Pro’s

• No limit on the number of pages it can crawl. The only limitations are from the machine it’s running on.
• The ability to check for broken external links.
• Ability to crawl JavaScript websites
• Can easily crawl a list of URL’s – great for when you’re doing Server Log Analysis
• Unlimited number of sites you can crawl
• Ability to identify large images

Con’s

• No schedule crawling (can be done but not easily)
• Runs on your own server/machine (recently launched the ability to run off an SSD drive – but I haven’t tested this)
• Historically comparison – while it can be done it excel, its a lot of manual work and the ability to write sumifs and vloopup statements. Its all manual.

DeepCrawl

Deepcrawl is a web-based application that has servers to host the bots for crawling. It’s a paid service of around £50 per month. It sounds a lot, but it means you don’t need to worry about slowing down or crashing your machine, etc. Everything is taken care of for you. For more information on Deepcrawl visit here.

Pro’s

• Schedule crawling.
• Change reports (from a previous crawl to the current crawl).
• Web-based (no dependence on your machine).
• Links to Google Analytics to give you even more data.

Con’s

• Doesn’t check to see if external links are active (as a standard feature, you need to check the box, see image below. use 1 credit per link checked)
• Limited crawl budget (need to increase monthly plan).
• Limited to the number of sites you can crawl. The basic plan only allows you to set up 5 projects a month and each project crawls one different website.

For more information on Deepcrawl visit here.

Overall, I do love both products and regularly use both products – it depends on the task I am wanting to do.

If you are unsure which is best for you – here is a video showing the key differences and where each one is best.

Summary Table

	Screaming Frog (Free)	Screaming Frog (Paid)	DeepCrawl Basic	DeepCrawl Agency
Cost	0	£149	£55 a month	£369 a month
Crawl Limit	500 pages per crawl	Unlimited*	100,000 URLs per month	1,500,000 URLs per month
Schedule	No	No	Yes	Yes
Comparison Reports	Manual	Manual	Yes	Yes
Active Projects	Unlimited	Unlimited	5**	30**

* Limited by computer resources

** Active projects are those which have a crawl run in the current billing period.

There are two other packages available from Deepcrawl. A Medium level plan and a bespoke plan.