Googlebot visits your website.
But how often?
What does it request?
Does your server respond successfully?
Are most requests for HTML pages, images, JavaScript, or something else?
Are crawling problems connected to server errors?
Google Search Console’s Crawl Stats report can help answer those questions.
Google describes the report as a record of Google’s crawling history for a site, including how many crawl requests were made, server responses, download size, response time, host availability, crawl purpose, file type, and Googlebot type.
There is also an important warning from Google:
The report is aimed at advanced users.
Google specifically says that websites with fewer than roughly a thousand pages generally should not need to worry about this level of crawling detail.
That is a useful place to begin.
For most small websites, the goal is not to maximize Googlebot activity.
The goal is to recognize genuine crawling or server problems when they occur.
Google’s official Crawl Stats report documentation
Where to Find Crawl Stats
Google currently places Crawl Stats under:
Search Console → Settings → Crawl stats
The report is available only for root-level properties, such as a Domain property or a URL-prefix property at the root of a host.
If you are looking at a narrower subdirectory property, you may not see it.
Start With the Summary
The main report includes:
Total crawl requests.
Total download size.
Average response time.
Host status.
Do not look at one metric in isolation.
Ask:
Did something materially change?
Total Crawl Requests
This is the number of requests Google issued for URLs or resources on your site.
Google explains that duplicate requests count individually, and unsuccessful requests can also be included.
Therefore:
More crawl requests ≠ better SEO.
A spike may reflect:
New content discovery.
Many redirects.
Image requests.
Resource requests.
Server problems.
Bot behavior unrelated to a ranking improvement.
Average Response Time
This measures how long your server takes to respond to Google’s crawl requests.
A temporary increase does not automatically mean disaster.
But if response time rises sharply and remains elevated—particularly alongside host problems or server errors—you may have something worth investigating.
Possible causes:
Hosting problems.
Resource exhaustion.
Plugin issues.
Database problems.
CDN configuration.
Large or inefficient responses.
Host Status
Host status is one of the most useful areas when diagnosing genuine crawling problems.
Search Console may show whether Google encountered issues involving:
Robots.txt availability.
DNS.
Server connectivity.
If your site suddenly has serious host availability problems, Google may be unable to crawl reliably.
That deserves attention.
Crawl Responses
The report groups requests by response type.
Common examples:
200 successful.
301/302 redirects.
404 not found.
5xx server errors.
Other response types.
The question is not:
Do I have any 404 requests?
Every mature website may accumulate some.
The better question is:
Did a response category change unexpectedly, and are important URLs affected?
Example: Sudden 5xx Spike
Imagine:
Monday: almost no 5xx responses.
Tuesday: a large spike.
Wednesday: still elevated.
That is more meaningful than noticing three isolated server errors.
Investigate:
Hosting logs.
WordPress errors.
Plugin changes.
Security tools.
CDN.
Traffic/load problems.
Redirects Are Counted Separately
Google says each server-side redirect request in a redirect chain can be counted separately.
If:
Page A → Page B → Page C
Google may record requests for each step.
That means unnecessary redirect chains can create additional crawl activity.
The right fix is not necessarily “reduce crawling.”
It is often:
Simplify the redirect.
File Types
Crawl Stats can break activity into file types such as:
HTML.
Image.
JavaScript.
CSS.
PDF.
JSON.
XML.
Audio.
Video.
Other.
This can help explain an unexpected crawl spike.
Maybe Google is not suddenly crawling 10,000 articles.
Maybe it is requesting many image or JavaScript resources.
Crawl Purpose
Google currently categorizes crawl purpose primarily as:
Discovery
Google is requesting a URL it has not crawled before.
Refresh
Google is recrawling something it already knows.
This can help interpret activity after publishing or updating a large amount of content.
Googlebot Type
The report can also show crawler categories such as:
Smartphone.
Desktop.
Image.
Video.
Page resource.
AdsBot.
StoreBot.
Other.
This is useful when one crawler type appears responsible for an unusual pattern.
Do Not Turn Crawl Frequency Into a Ranking Metric
A common SEO mistake is:
Googlebot visited more, so Google must like my website more.
Crawl activity and ranking are not the same thing.
Google crawls pages to discover and refresh information.
Search ranking depends on much more than how many crawl requests occurred.
Do not make business decisions from crawl volume alone.
Small Sites Usually Do Not Need Crawl-Budget Optimization
This is worth repeating because Google says it directly.
If your site has fewer than about a thousand pages, Google says you generally should not need to use Crawl Stats or worry about crawling at this level.
For a typical small WordPress site, focus on:
Important pages accessible.
Server reliable.
Sitemap accurate.
Internal linking sensible.
No accidental robots blocking.
No huge duplicate-URL problem.
That is usually more useful than trying to “optimize crawl budget.”
When Crawl Stats Becomes Useful
Use the report when:
Google Seems Unable to Reach Important Content
Check host status and responses.
You See Many Server Errors
Investigate response groups.
A Site Migration Changed Crawling
Compare before and after.
Crawl Requests Suddenly Spike
Determine what file or Googlebot type changed.
A Large Section Is Not Being Refreshed
Investigate discovery, sitemaps, internal links, and site structure.
Crawl Stats Is Not the Page Indexing Report
These reports answer different questions.
Crawl Stats
How is Googlebot interacting with my server?
Page Indexing
Which known pages are indexed or not indexed?
A page can be crawled and still not be indexed.
Likewise, a crawl spike does not mean hundreds of new pages were added to Google’s index.
Use Representative Example URLs
Google says the detailed examples inside grouped crawl data are representative, not a complete list of all requests.
If you click:
5xx responses
you may see sample URLs.
Use them to understand the pattern.
Do not assume the examples represent every affected URL.
Compare Crawl Problems With Search Traffic
Suppose server errors spike on August 1.
Search traffic falls on August 3.
That timing deserves investigation.
It does not automatically prove causation.
Use the broader workflow in How to Diagnose a Drop in Google Search Traffic Before You Change Your Website.
Look at:
Performance data.
Indexing.
Crawl status.
Site changes.
Seasonality.
Google-confirmed incidents.
Do Not Block Resources Just to Reduce Crawl Numbers
You may notice Googlebot requesting:
Images.
CSS.
JavaScript.
Do not automatically block them.
Google may need page resources to understand and render content.
Only block resources when you understand the implications.
Watch for Robots.txt Availability
Crawl Stats can help reveal robots.txt availability problems.
Google notes that crawl attempts affected by insufficient robots.txt availability can influence crawl reporting.
A missing or intermittently inaccessible robots.txt file can therefore be relevant even when your actual rules are simple.
Use Sitemaps for Important Content Discovery
If you publish new content and Google is not discovering it, confirm your sitemap contains the important canonical URLs.
A sitemap does not guarantee indexing.
But it provides Google with a useful discovery signal.
A Simple Monthly Crawl Review
For a small website, monthly or problem-triggered review is usually enough.
Record:
Total Crawl Requests
Average Response Time
Host Status
5xx Change
404 Change
Top File Type
Important Observation
Then ask:
Is there a real problem requiring action?
If no:
Close the report.
Example
Suppose:
Requests increased 35%.
Host status is healthy.
5xx remains near zero.
Response time is stable.
Increase comes mainly from image crawling after publishing many image-heavy articles.
Possible conclusion:
No immediate technical problem.
That is better than assuming increased crawl activity means success or failure.
When to Contact Your Host
Consider hosting support when you see:
Sustained 5xx failures.
DNS/connectivity problems.
Major response-time deterioration.
Host-status warnings.
Repeat crawl failures affecting important content.
Bring evidence:
Date.
Response type.
Example URLs.
Response-time pattern.
That makes the support conversation more productive.
Conclusion
The Search Console Crawl Stats report is useful when you need to understand how Googlebot is interacting with your server.
Use it to investigate:
Host availability.
Response errors.
Slow responses.
Unusual crawl spikes.
File-type patterns.
Discovery versus refresh crawling.
But do not turn crawl counts into a vanity metric.
For most small websites, you do not need to “maximize crawl budget.”
You need a site that is accessible, technically stable, logically organized, and easy for Google to crawl when it needs to.