Blog / Technical SEO /
A careful way to troubleshoot sudden crawling changes
The crawl request graph drops off a cliff, and the first explanation in the room is “Google is crawling us less”. Maybe. Or maybe the log filter changed, the time zone shifted or a deployment made the server slow.
Before reacting, check that the observation is real, what changed and which URLs are affected. Crawling varies for plenty of reasons: site response, URL generation, reporting windows. A careful investigation separates a dashboard signal from its cause.
Validate the measurement
Check the date range, time zone, server coverage and bot identification method. If you use logs, confirm requests attributed to Googlebot are verified properly and the relevant hosts are included. Compare like with like, and note deployments, traffic peaks and outages. Search Console crawl statistics and server logs give different views, and neither is a complete account of indexing.
Find the affected pattern
Segment requests by host, path, response code, user agent and URL parameters. Compare important pages with low-value or duplicate routes. Check whether a release changed robots rules, canonicals, sitemaps, internal links or response times. Look for spikes in errors and repeated requests to the same paths before touching any access rules.
Respond to evidence
Fix clear server failures or unwanted URL explosions, then watch the same segments. Don’t block paths reflexively, because you might hide useful content or directives. Keep a timeline of the change and the next review.
Resist the first story
Early in an investigation, one explanation feels obvious and everything starts to confirm it. I try to write down two or three possible causes before I look at the data, and list what each would predict. A server slowdown, a robots change and a URL explosion leave different fingerprints, so the evidence can choose between them.
If nothing is clearly wrong, it’s fine to say “no action needed, we’ll recheck on this date”. A calm note beats a change made to feel productive.
Check the boring causes first
Before the clever theories, rule out the dull ones. Did the site go down or slow down? Did a deployment change robots.txt, redirects or the sitemap? Did a CDN or security setting start challenging bots? Did the log source change, so the numbers simply stopped including a host?
I check these first because they’re quick, and because they’re fixed by a person with the right access rather than by a long investigation. Ask whoever manages hosting and release notes what changed in the window, and keep their answer with your timeline.
Finish by writing the conclusion in one honest sentence, even if it is only “we found a slowdown on one host and fixed it” or “we found nothing and will recheck on Friday”. A clear note stops the same question being investigated again next month.
Apply the review to one real case
Define what a finished investigation changes for the people using the site, not just what work the team did. Use the same scope and definitions across examples and record what you excluded, because one noisy segment can make a pattern look universal.
Follow one real URL from the log entry to its live response and rendered state. Check what a visitor and a crawler can each understand or do. Before acting, confirm which source can verify the fact and whether it’s current, since an old export may describe an earlier configuration.
Check the outcome in the live environment, since apps, caching, CDN rules or theme behaviour can undo an approved fix. When sources conflict, compare timestamps, definitions and ownership before choosing a value.
Keep logs in an approved place and out of casual chat. If the information is incomplete, a small reversible change beats a sitewide rule. Allow exceptions only for real problems and write down their boundaries. When you close the work, check nearby routes that depend on the same facts and make sure someone knows how to restore the previous state.
A worked example: the host that went missing
Say the crawl graph falls sharply on a Monday. Before touching any robots rules, you compare the log source with last week’s. A subdomain that serves a regional storefront is no longer in the export, so the totals dropped without Google changing anything. The investigation ends there: repair the log feed, note the gap in the timeline and recheck next week. If the host had been present and requests had still fallen, you’d move on to response times and release notes. The point is the order. Confirm what the numbers cover, then look for causes, and only then consider changing anything on the site.
Related reading
- What DNS changes can affect during a website move
- What to review when a site changes its rendering strategy
- Technical SEO: redirect maps for catalogue changes
What changed in the affected URL set just before crawling shifted?