COVID-19 Business Continuity Insights and Digital Marketing Tips

Take Me There

Making the Most of Screaming Frog’s Features in 2020

Many factors play key roles in a website’s search engine ranking, including keywords and backlinks. But a site structure that makes crawls simpler for search engine bots is also...

Josh Evora
Josh Evora October 20, 2020

Many factors play key roles in a website’s search engine ranking, including keywords and backlinks. But a site structure that makes crawls simpler for search engine bots is also essential.

Indexing allows a website to show up in search results, and to make that happen, bots should be able to effectively crawl your site. Looking for an unindexed site is next to impossible (unless a recent indexing glitch has happened), and it would make the site practically non-existent since it can’t be found anywhere on the web. 

And that’s why a site structure that enables efficient crawls is a vital part of a successful on-page SEO strategy.

SEO experts use various tools to ensure that search engine bots can crawl and index websites. Screaming Frog is one of these tools — it’s a web crawler that helps you audit your site and gives you insights on its performance. 

The software might be tricky to use if you’re starting out, but don’t fret. Our in-depth Screaming Frog guide will help you explore and use the tool to your advantage.

What is Screaming Frog?

Screenshot from Screaming Frog on YouTube

Screaming Frog works like Google’s crawlers: it lets you crawl any website, including e-commerce sites. But this SEO spider tool takes crawling up by a notch by giving you relevant on-site data and creating digestible statistics and reports. With simpler site data from Screaming Frog, you can easily see which areas your website needs to work on.

The Screaming Frog software can help you perform the following tasks for your SEO efforts:

And of course, you can do a lot more with Screaming Frog. We’ll cover these extra features in another section.

Getting Started with Screaming Frog

Installing the software

Before performing any site crawls through the Screaming Frog SEO spider tool, you’ll need to install the software on your computer. You can download the tool from Screaming Frog’s website.

The software works on Windows, Mac, and Linux. Here’s how to install Screaming Frog on any of these systems:

Windows

  1. Find the Screaming Frog installer in your Downloads folder.
  2. Double-click on the installer.
  3. Click “Yes” on the User Account Control screen to continue installing the software.
  4. Choose your installation type, then click the “Install” button.
  5. Once the software finishes installing, click “Close”.

macOS

  1. Go to the Downloads folder in Finder.
  2. Double-click on the installer.
  3. A new window with the software’s icon and Applications folder will then appear on your screen. Click the Screaming Frog icon and drag it to the Applications folder.
  4. Close the window.
  5. Go to Finder and look for the “ScreamingFrogSEOSpider” name in the Devices list.
  6. Click the eject icon next to the installer name to finish the installation.

Linux

You can use the Ubuntu user interface or command line function to install Screaming Frog on your computer.

Ubuntu User Interface
  1. Double-click on the software’s .deb file.
  2. Choose “Install”, then enter your password.
  3. Accept the ttf-mscorefonts-install license before installing Screaming Frog.
  4. Wait until your computer finishes installing the software.
Command Line

1. Type the following command in an open terminal window:

Screenshot from Screaming Frog

2. Enter your password.

3. Type Y to continue installing the software and accept the ttf-mscorefonts-install End User License Agreement (EULA).

Do keep in mind that you’re installing a free version of Screaming Frog. The free version lets you crawl a maximum of 500 URLs, and you’ll need a license to crawl more websites. You can buy a software license on the Screaming Frog website.

To start using Screaming Frog’s premium version, go to the License menu. Click on the “Enter License” option and type in your license key. You should see a dialog box that shows the license’s validity and its expiry date.

Studying the user interface

Screenshot from Screaming Frog on YouTube

After installing the Screaming Frog tool, it’s best to study the software’s menus, options, and settings. Let’s take a closer look at the different menus and settings that you can use.

File

This menu lets you save a crawl as a file and see your last six (6) crawls you performed if you haven’t saved them yet. Plus, the File menu will let you set default settings for the software.

Configuration

If you’re after custom crawl settings, Screaming Frog’s Configuration menu allows you to set and adjust those options. You can use the options below to customize your crawl settings.

Bulk Export

As its name suggests, the Bulk Export menu lets you export multiple URLs. Screaming Frog will accept links with various site elements, including:

Reports

The Reports menu creates downloadable crawl overviews and on-site data reports. You can use these reports in your SEO audits.

Sitemaps

With this menu, you can create your website’s sitemap. You’ll learn more about constructing sitemaps in the latter part of this Screaming Frog SEO tutorial.

Setting up your device’s memory & storage

Screenshot from Screaming Frog on YouTube

Site crawls apply to big and small websites. But crawling larger websites consumes more memory and processing power. With Screaming Frog, you can allot a certain amount of your device’s memory for crawling websites. The software has database and RAM storage modes for this purpose.

Screaming Frog’s database storage mode is ideal for users with solid-state drives (SSDs). To use the mode, follow these steps:

  1. Click on the Configuration menu.
  2. Select the System option, then click Storage.
  3. Select the Database Storage Mode option.

Meanwhile, users without SSDs can settle for the RAM storage mode, with a default setting of 1GB RAM for 32-bit computers and 2GB RAM for 64-bit devices. A lower RAM prevents freezing and crashes while crawling with Screaming Frog.

But if you want to allocate more RAM for crawls, you can change the default RAM setting to a higher number. The changes will apply after you restart Screaming Frog.

Configuring settings

Screenshot from Screaming Frog on YouTube

Screaming Frog already follows default settings for site crawls. But you can tweak these settings and collect specific data using the software’s numerous tools. By using these tools, your crawls will take up less time and processing power. You can customize your settings from the Configuration menu.

Customizing windows and columns

Screenshot from Screaming Frog on YouTube

Along with customizing settings, Screaming Frog lets you tweak windows and columns for easier access. When you run the software, you’ll see three (3) windows, and the right-side column allows you to access SEO elements and filters. On the other hand, you’ll find a window directly below the main that displays specific web page data.

Adjusting window sizes is a great way to customize your view in Screaming Frog. You can resize windows by dragging them to your desired size. Plus, the software will let you customize columns any way you want. 

Clicking and dragging a column will move it to your desired position, while clicking a column will sort your data. For instance, if you want to sort your columns by number, you can simply click on the column and the software will arrange the numbers from highest to lowest.

Using Screaming Frog for Basic SEO Tasks

Screaming Frog stands out from other SEO tools because it lets you search, filter, and customize your crawls. It also offers a wealth of benefits, from analyzing site enhancements and errors to filtering the way your webpages show up. 

From website crawls to sitemap creation, here’s how to perform some SEO tasks in Screaming Frog.

Crawling a website

Performing link audits

Performing content audits

Creating an XML sitemap

Other Useful Screaming Frog Features

Most of the time, you can just crawl websites with Screaming Frog’s basic tools. But as you explore the software, you’ll find more helpful features for site crawls. Here are some of its other features you might have missed:

Improve Search Rankings with In-Depth SEO Audits

SEO audits help you identify possible changes to your website’s various elements. This way, you can climb to the top of the search ladder and surpass your competitors. Plus, you can use insights from your audits to improve UX and other technical elements of your site.

With a reliable partner on your side, you can get a closer look at your site performance and find out how you can improve it further. Get in touch with Growth Rocket today and we’ll help you slowly make your way to the top of the search rankings.

More From Growth Rocket