Screaming Frog Site Crawl

Let's Go!

Need help?

Schedule a Jam Session today. We'll help you walk through an SOP, Template, Client Setup, or anything else you need!

Let's Jam

included:

Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Add to my favorites

0

Add to my favorites

0
Download Template
View SOP

Template

Download or Make a copy to your Google Drive.

Save in Account

0

Save in Account

0
Download TemplateDownload PDF version
Click to Download

SOP

Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Save in Account

0

Save in Account

0
Download PDF version
  1. Load Custom Crawl Configuration configuration using  File > Configuration > LoadLink to crawl configuration file: Download
  2. Apply Custom Search Settings using Configuration > Custom > Search.
    Confirm each filter row is set to "Contains", the copy and past the values under "Custom Search Settings" in this sheet. Click OK
  3. Change the robots.txt configuration using Configuration > robots.txt > settings. Select "Ignore robots.txt". Click OK
  4. Enter the {CLIENT} URL into the box at the top and click Start.
  5. Confirm the URL tab is selected, and click Export.
  6. Confirm the URLs in the exported report are sorted A > Z. Copy the list of URLs and paste into the URL column of the Tag Coverage tab.
  7. At the right of the same toolbar is a downward arrow icon that bring up a menu with more report options. Click on this icon, and select Custom
  8. On the Custom report screen, you will see a sub-menu labeled Filter: with a button to the right labeled Export. Export the first report ( Contains: fbq\('trackj\'\, \'Page.iew\'\) ).
  9. Copy the entire sheet of exported results, and paste into the custom_search_1 tab.
  10. Repeat the previous step for the remaining custom search reports, utilizing the appropriate custom_search tabs to collect data.
  11. Create a directory tree graph to visualize site architecture using Visualizations > Directory Tree Graph.
    Paste a screenshot of this graph into the Structure Visualizations tab.
  12. Create a force-directed crawl diagram to visualize internal linking relationships using Visualizations > Force-Directed Crawl Diagram.
    Adjust the node positions, node separation, node overlap, and node size to attain the highest degree of readability possible.
    Paste a screenshot of this diagram into the Link Visualization tab.
  13. Select the Page Titles tab, and confirm the Filter: is set to "Over 600 Pixels".
    Click Export if any pages are listed, and paste the entire exported sheet into the raw_title_tags tab.
  14. Update the Filter: setting to "Missing". Export the report if any pages are listed.
    Beginning with row 3 on the exported report, paste these values below the values that were added to the raw_title_tags tab in the previous step.
  15. Select the Meta Descriptions tab, and confirm the Filter: is set to "Over 160 Characters".
    Click Export the report if any pages are listed, and paste the entire exported sheet into the raw_meta_descriptions tab.
  16. Update the Filter: setting to "Missing". Export the report if any pages are listed.
    Beginning with row 3 on the exported report, paste these values below the values that were added to the raw_meta_descriptions tab in the previous step.
  17. Select the Images tab, and update the Filter: to "Over 100 KB". Click Export.
  18. In the exported file, sort the Size (bytes) column from Z > A.
    Starting with the header, copy and paste all rows that have values exceeding 200,000 in the Size (bytes) column into the raw_large_images tab.
  19. While on the Images report in Screaming Frog, select the Inlinks tab at the bottom.
    For each image over 200 KB, copy the From URL and paste it next to the corresponding Image Address in the Images tab of this spreadsheet.
  20. Update the Filter: setting to "Missing Alt Text". Click Export if any images are listed.
  21. While on the Images report in Screaming Frog, select the Inlinks tab at the bottom.
    For each image missing alt text, copy the From URL and paste it next to the corresponding Image Address in the Images tab of this spreadsheet.
  22. Select the H1 tab, and confirm the Filter: is set to "Missing". Export the report if any pages are listed, and paste the entire exported sheet into the raw_no_h1 tab.
  23. Update the Filter: setting to "Multiple". Click Export if any pages are listed, and paste the entire exported sheet into the raw_multiple_h1 tab.
  24. Select the Internal tab, and filter the Status Code column Z > A.
    For any pages with a status code other than 200, copy directly from the Address column, and paste under the Address heading in the Internal sheet.
    Repeat the same task for the Status Code column, now utilizing the Status Code heading on the same sheet.
  25. Select the Inlinks tab from the bottom of the screen in Screaming Frog.
    Copy the all entries in the From column, and paste next to the corresponding pages that you just added to the Internal sheet.
  26. Select the External tab, and filter the Status Code column Z > A.
    For any pages with a status code of 301 or 404, copy directly from the Address column, and paste under the Address heading in the External sheet.
    Repeat the same task for the Status Code column, now utilizing the  Status Code heading on the same sheet.
  27. Select the Inlinks tab from the bottom of the screen in Screaming Frog.
    Copy the all entries in the From column, and paste next to the corresponding pages that you just added to the External sheet.