Screaming Frog SEO spider is my tool of choice when I need to crawl, analyse and audit a site's technical SEO.
If I somehow ended up in a situation where I could only keep one of my many tools for SEO, this little program would, without a doubt, be the one.
The Screaming Frog SEO Spider is a small SEO tool developed by Screaming Frog a search marketing company based in England. Screaming Frog will spider that websites’ links, images, CSS, script and apps from an SEO perspective. It returns key onsite elements for SEO, presents them in an orderly manner, divided in tabs by type, and gives you the opportunity to filter the data for common SEO problems. Just install this locally on your PC, Mac or Linux machine and you can crawl any site you want.
This is my go to tool in so many different situations. It works brilliantly for a quick audit of a website. This is usually the first thing I do when someone asks if we can help with their onsite optimization, a quick crawl with Screaming Frog gives us an overview of the structure of the site and what issues are present today. Without this tool, this would have been a time consuming manual task, where every site would have had to be visited individually.
The picture above is from one such crawl for a potential customer, and it is easy to see that there are several issues that need to be resolved. First of all, there are eight links to pages that does not exist and therefore return a 404-response code. In addition, you can see in the list on the right side that there are 425 duplicate page titles, and 425 duplicate meta descriptions. The five minutes it took to gather this data and review it, tells us that there is a lot of work to do for this client.
Screaming Frog gives us the opportunity to easily look for:
So to sum it up the Screaming Frog SEO spider is my tool of choice when I need to crawl, analyse and audit a site's technical SEO. It is just as good if you just need a quick overview of a small site or you have to analyse a site with thousands of pages. You can download a free version over at Screaming Frog’s webpage. The free version comes with some restrictions, you can only crawl 500 URL’s in one crawl and you cannot access all the configuration features. Nevertheless, it is worth a download so you can try before you buy. I tried and I bought.