The SEO’s Toolbox: Screaming Frog

Screaming-Frog-LogoWelcome ladies and gentlemen, boys and girls to another exciting instalment of “The SEO’s Toolbox.”

Today I’m delighted to be joined by Dan Sharp of Screaming Frog, one of the most brilliantly named SEO tools/agencies around. Dan has been super cool and offered to hang around the blog today to answer any questions you guys may have about Screaming Frog not covered in the post.

I’m constantly surprised with the creative uses SEO’s come up with for this tool so I’m definitely looking forward to this one. Huge thanks to Dan for getting involved!

First of all welcome and thank you for agreeing to do this interview. To kick off I’d like to know a little more about you and how you first became involved with Screaming Frog?

“Thanks for inviting me Gaz, it’s a pleasure. I’m Director at Screaming Frog, who are a search marketing agency based in Henley-on-Thames in the UK. I think like a lot of people, I fell into the world of search completely by accident. It was my very first job straight from university, which was a long-time ago now! I have worked agency side and as a freelance SEO consultant, before founding Screaming Frog around two and half years ago. You can read more about how we came up with the name, the remarkable team we now have and our story over here.

As an SEO I was a little frustrated with the tools available to help perform onsite audits and provide a base for SEO recommendations for clients. I had used Xenu for years (which I still love!), but it didn’t really do quite enough from an SEO perspective, as obviously it wasn’t built for SEO.

So around three years ago now while working as a consultant, work started on the ‘Screaming Frog SEO spider’ an advanced desktop SEO crawler. I wanted it to be a desktop crawling application built specifically for an SEO who knows what they are doing with the data. At the time it was being built, it was actually only going to be used by me internally, for my own client SEO projects. But, then as Screaming Frog the agency was formed and we grew, we made the decision to release publically.”

Tell us more about Screaming Frog itself?

“The Screaming Frog SEO Spider is a desktop crawling program which spiders websites from an SEO perspective. It’s available for Windows, Mac and Ubuntu for free up to 500 URLs and we have a licenced version which opens up the crawl limit and configuration options for £99 p/a.

Essentially the SEO Spider collects key onsite SEO data which allows you to quickly analyse and audit a website. It spiders all URLs of a website and collects data such as response codes, page titles, meta descriptions, headings, in links, out links, as well as things like anchor text, alt text, robots directives and canonical elements etc. This then easily allows you to find technical issues, such as client or server side errors, redirects, duplicate pages and so on. It can be really easy to miss the odd meta refresh here, 302 there or X-Robots-Tag http implementation gone awry purely by manual checks. It also obviously takes a lot of time, particularly on enterprise sites, so the SEO spider provides the support needed.

The tool also allows you to export all the data to Excel, so you slice it and dice it however you please. It doesn’t tell you how to do SEO of course, it just gives you the data to make informed decisions. It was built very much with an experienced SEO in mind who needs to get down and dirty into a proper onsite audit.”


If there anything exciting coming up in the next few months? What does the future have in store for Screaming Frog? Do you have a vision of where you see Screaming Frog heading in the short to medium term?

“We have a huge list of features we still want to add to the tool and there are a variety of new features in development which will be coming out over the next few months. We will be providing support for rel=”alternate” hreflang and rel=”next” & rel=”prev” as a couple of examples. We get a lot of amazing feedback from the SEO community and many of these requests get put into development as well. If you have any ideas, just let me know as we actually listen, consider everything and often implement them!

In the longer term we are working on something behind the scenes which will significantly improve crawling at scale with automation. I won’t go into any more detail than that, but it’s already very exciting.”

The way I tend to find out about the best tools (both new or old) is via blogs or presentations from other SEO’s who’ve often been using them for a while. Is there any post or slide decks featuring your tool you’d recommend?

“There are some amazing blog posts which provide some very cool new ways of using the SEO Spider. I am still surprised today with the new ways that SEOs are using it. Here are a few which we love –

Combining AWR and Screaming Frog to Spot Quick Wins – from Jon Quinton

8 Alternative Ways To Use Screaming Frog for SEO – from Craig Bradford

Checking Google Analytics Is Installed On Every Page with Screaming Frog – from Kev Strong

Visualize your Site’s Link Graph with NodeXL – from Aleyda Solis

Low Hanging Fruit: Link Building with Screaming Frog – & Scaling Internal Link Building with Screaming Frog & Majestic – from Ethan Lyon & John-Henry Scherck respectively, both from SEER.

There’s many more as well!”

How about some headlines, give me a couple of interesting facts about Screaming Frog?

“We’ve had well over a hundred thousand downloads of the tool now which is fairly remarkable, considering it was a tool that was built originally quite selfishly for one person by a single person. Our user and customer base has grown substantially over the past year in particular as well, we have many of the biggest SEO agencies in the world using it, as well as large worldwide brands, right down to freelance consultants. The growth has involved zero marketing budget, simply word of mouth by passionate SEOs in the industry.”


What about pricing models?

“The SEO Spider is actually free to download and use, but it’s limited to 500 URLs and the configuration options are restricted. You can purchase a licence for £99 p/a, which removes the 500 URL crawl limit, provides access to all the configuration options, allows saving of crawls, access to the custom source code search feature and of course full technical support for the software. We will probably review our pricing model this year, it’s actually worked really well, but I think it could be improved further.

Thanks again for including us in your ‘The SEO’s toolbox’ series!”

Thanks again to Dan for taking the time out to answer these questions and giving some insight into Screaming Frog. Don’t forget to drop a few questions for him into the comments below 🙂

Post by Gaz Copeland


  1. Anthony Pensabene says:

    Great job as a producer on the series, Gaz. Thanks for the resources and review. I got to see Ethan and John-Henry’s presentation firsthand at the Philly Grail meetup a few months ago – it was most solid.