Blog / Technical SEO /
When to use log analysis in an SEO investigation
A traffic investigation says a page is missing from the reports. What it can’t say is whether the page was requested, returned an error or was never discovered. Analytics only sees people whose browser ran the tag. Something else has to see everything else.
Server logs can answer questions about requests and responses that analytics can’t. They need care, though. Logs are technical records, not a direct measure of human visits or search performance.
Use logs when the question fits
Log analysis helps when you need to check whether verified search crawlers request important URLs, how often a route returns errors, or whether parameter patterns eat up requests. It’s useful during migrations, crawl investigations and incident diagnosis. It won’t tell you why a page ranked, or whether a person found it useful. Start with a concrete question before you ask for access to a huge log archive.
Prepare and interpret data carefully
Confirm which hosts, time zones and fields the logs cover. Identify bot requests with a reliable method, group paths carefully, and account for query strings and redirects. Protect IP addresses and other personal data according to policy. Sampling, retention limits and infrastructure layers can hide parts of the request path, so write those limits down before drawing conclusions.
Connect findings to action
Compare logs with Search Console, a crawl and the site’s release timeline. Investigate the pattern, then confirm a fix through later requests and live checks. Don’t change robots rules because of a single chart.
Know when not to use logs
Logs are heavy. Getting access, cleaning the data and verifying bots takes real time. If a crawl or Search Console report already answers your question, you probably don’t need them. I’d reach for logs when the question is about what a crawler actually requested and got back, and I’d skip them when the question is about what a visitor did.
A good rule of thumb: write the question in one sentence, and name the other source that could answer it more cheaply. If there isn’t one, logs are worth the effort.
Handle the data responsibly
Logs contain IP addresses and URLs that may carry personal information in query strings. Limit who can open them, work from extracts where you can, and delete working copies when the investigation ends. Say in your notes which fields you used and which you ignored.
I also write down how I identified the crawler. Matching a user-agent string alone isn’t enough, because anyone can send that string. Use the verification method your hosting or search provider documents, and say in the report that it was applied, so the next reader knows how far to trust the counts.
Apply the review to one real case
Define what a finished investigation changes for the people using the site, not just what work the team did. A busy day or a single host can make a pattern look more universal than it is.
Follow one real URL from the log line to its live response and rendered state, and check what a crawler and a visitor can each understand or do. An old export may describe an earlier configuration.
Check the outcome in the live environment, since apps, caching or CDN rules can undo an approved fix. When two sources disagree, compare timestamps, definitions and ownership before choosing.
Keep log extracts in an approved place and avoid personal data in shared notes. If the information is incomplete, a small reversible change beats a sitewide rule. When you close the work, check nearby routes that depend on the same facts, and make sure someone can restore the previous state.
A hypothetical: checking redirects after a migration
Say a store has just migrated and a set of old product URLs should now redirect. A crawl shows the redirects in place, but Search Console is slow to reflect them. The question is narrow: are search crawlers still requesting the old URLs, and what do they get back? Logs answer exactly that. Filter to verified crawler requests for the old paths, group them by response code, and see whether they end in a single redirect to the right page or in chains and errors. If the redirects resolve cleanly, you have evidence to leave things alone and recheck later. If not, you have a list of paths to fix.
Related reading
- A careful way to troubleshoot sudden crawling changes
- How to review JavaScript errors that may affect search access
- A site migration risk register for a small team
Which unresolved SEO question would server logs answer more clearly than analytics?