So, Google today launched an all-new update to its ever-evolving Search Console product. A brand new “Crawl Stats Report” that promises SEOs more data and insights. How can SEOs make use of it. Let’s take a look.
How to Make Use of Google All-New Crawl Stats Report
Here’s the official tweet from Google Search Central.
What is Google Crawl Stats Report
The Crawl Stats report shows you statistics about Google’s crawling history on your website. For instance, how many requests were made and when, what your server response was, and any availability issues encountered. You can use this report to detect whether Google encounters serving problems when crawling your site.
So, looks like it’s an amazing tool for SEOs to find out what’s wrong with your website. And of course, help fix those things that are causing Google not to crawl your website properly.
For beginner SEOs, let’s take a step back.
What is search engine crawling?
Crawling is the first out of the three primary functions of search engines. The functions are Crawling, Indexing, and Ranking.
- Crawling: This is the first step where a search engine discovers a new link/URL from a website and scans the content.
- Indexing: In this step, a search engine stores and organizes the content discovered during the crawling process.
- Rank: In this step, a search engine chooses parts of the content indexed that match a search query in an ordered list format (Most relevant to least relevant).
What does Crawl Stats show?
As of now, crawl stats provides the following stats.
- Total crawl requests to the entire website in the chosen timespan.
- Total downloaded size in Bytes.
- Average response time in Microseconds.
Total crawl requests
A total number of crawl requests to your site, in the time span shown. Duplicate requests to the same URL are counted.
Total downloaded size
The total size of all files and resources downloaded during crawling, during the period shown. If a resource is already cached from a previous crawl it is not counted. Bytes include file HTML, associated image bytes, script files, and CSS files.
Average page response time
Average page response time for a crawl request to retrieve the page content. Does not include retrieving page resources (scripts, images, and other linked or embedded content) or page rendering time.
Source – Google
Google Search Console will also show us the child domains in your website and the issues associated with them if any.
Clicking on the items will reveal the details.
It also shows the crawl requests breakdown by the response, file type, purpose, and Google bot type.
Crawl Stats by File Type
This metric shows the split between different file types and if there are any unusual issues that need attention.
Also, this is an excellent way to figure out if there are issues with any particular file type you’re hosting, that are causing a larger issue – like page speed and load times.
For example, if there are several/any JS files that is taking an unnatural amount of time to crawl. Perhaps from a third part site like (Facebook or Google AdSense).
Click through on each file type to narrow down on the issue.
Crawl Stats by Response
Crawl stats by response is probably the best way to find out if there are common issues with your website.
Like 404 errors (broken links) for example.
It shows five types of responses. From OK (200), Not Modified (304), Moved Permanently (301), Not Found (404) to Page could not be reached.
A sudden spike in any of these could mean there’s an issue gone undetected and you need to fix it asap.
Nice!
To summarise, here’s how SEOs can make use of Google’s Crawl Stats report.
- Evaluate site health check in one glance.
- Find out what resources are causing page load issues. (using Crawl Stats by File Type)
- Find out issues with broken links, 301 redirects, etc (using Crawl Stats by Response)
- Find out health issues of child/sub domains separately.
- Find out if any part of the website is unavailable for crawling. (using Host Status)
I think Crawl Stats is going to be the go-to place for SEOs to do a reliable site health check, without using third-party tools.
It’s nice to see Google Search Central team come up with these useful updates. Long time coming!
Leave a Comment
You must be logged in to post a comment.