# Reading server logs for questions a crawler tool cannot answer

URL: https://ekom.blog/reading-server-logs-for-questions-a-crawler-tool-cannot-answer/
Published: 2026-02-05
Author: Ekom
Topics: Technical SEO

> Server logs can show which paths were requested and what the server returned, but they cannot tell you whether a page was indexed. Start with one question and record the limits of the evidence.

The crawler says the page returned fine. Your browser says it loaded. And yet the search traffic still doesn't add up, and the developers swear requests behaved differently between releases.

A crawl tool is great for links and responses, but it only shows what its own requests could see. Server logs answer a different question: which URLs were requested, by whom, and what did the server send back at that moment?

Logs aren't a ranking report. They're a record of requests, often noisy, with retention limits and privacy strings attached. I only open a big log file once I've written down one clear question.

## Begin with the question, not the export

A log review works best when it tests a hypothesis. Maybe you want to know whether an important template is being crawled after a release. Maybe requests are still reaching old URLs that should have been retired. Maybe a crawl says a set of pages is inaccessible and you want to see if search crawlers request them at all.

Write the question in plain language, then decide what evidence would support or weaken it. "Are search engines crawling our site?" is too broad. "Did Googlebot request the new product template after the deployment?" is better, though it still needs a date window, a way to identify the requests and confirmation that the logs are complete for that period.

## Understand what the log actually records

A web server log commonly records a timestamp, requested path, response status, user agent, client address and other request details. The exact fields depend on the hosting or delivery setup. Requests may pass through a CDN, proxy or application layer, so the file you receive may not show the whole journey.

Before reading any rows, ask whoever manages the infrastructure where the log came from, which time zone it uses, how long it's kept and whether it includes requests served from cache. Check whether fields are redacted or transformed. That context decides what you can conclude. It's the first thing I ask for.

A user-agent that says "Googlebot" isn't proof Google made the request. A log records a claimed identity. Validating it may need checks against Google's published guidance or a trustworthy platform report. Don't label every crawler-looking request a search engine because the name sounds familiar.

## Filter carefully and keep the rules visible

Big files are hard to inspect, so filter. Record the filters you applied: date window, domain, path pattern, response codes and how you identified bots. Keep the original file in the approved storage location, and don't upload sensitive logs into a tool without permission.

Start with a small sample. Check whether timestamps parse correctly, whether query strings are present, and whether encoded characters or trailing slashes create separate versions of what looks like the same route. A filter that strips too much can combine distinct pages; one that is too narrow can hide the pattern.

If you're using a spreadsheet or script, keep the raw fields and add derived columns separately. You'll spot a parsing mistake before it becomes an SEO conclusion. If a request path contains personal information, follow the site's data handling rules and minimise what you keep or share.

## Use logs to answer crawler questions

For an indexing investigation, compare requests with a known list of important URLs. Look for patterns by template, not only one-off paths. Did the crawler request the new URLs? Did it continue to request redirected or retired routes? Are some areas receiving repeated requests while others appear absent from the sample?

Absence from a log doesn't prove a URL was never crawled. The file may be incomplete, the request may have been served elsewhere, or the window may be too short. And a request doesn't prove a page was indexed or valued. Logs show server activity, not the crawler's final decision.

For a migration, compare the old and new path patterns around the launch date. Look for requests to old URLs and check whether responses match the redirect plan. A recurring 404 may be an overlooked internal link, an external link or simply a request for a URL that never existed. Use the referer field only with care; it may be missing, truncated or not reliable enough to establish the source.

## Pair logs with the evidence they cannot provide

Logs are stronger when combined with other sources. A crawl tells you what a crawler tool could reach from your starting points. Search Console can show reported indexing and performance information. A sitemap can show the URLs the site claims are important. Deployment notes explain what changed. Together, these sources help you ask a narrower follow-up question.

For example, if the sitemap contains a new group of pages but the log sample has no matching requests, check whether the file is complete, whether the pages are linked internally and whether the request path is being served through a different layer. Do not jump immediately to “Google is ignoring these pages”. The evidence has not established that.

Keep the conclusion in proportion. "In this log sample, we found requests to the new category routes after the release" is safe. "Google has indexed the new categories" needs different evidence. I'd rather say less and be right.

## Make the result useful to the next person

A good log investigation ends with a short note: the question, source and date window; how requests were identified; the pattern observed; limitations; and the next check. If the work leads to a technical change, record the affected paths and who will validate the response after release.

Don't leave a huge export as your conclusion. A colleague should understand what you did without repeating the whole analysis. Keep raw data only as long as your organisation's policies allow, and limit access to people who need it.

## Related reading

- Checking whether a site search form exposes crawlable URLs

- How to review a site’s response to a crawler spike

- How to investigate unexpected indexable query strings

Before you open another log file, name the one crawler question you need it to answer. What evidence would change your next technical action?

---

From Ekom (https://ekom.blog/). More: https://ekom.blog/llms.txt
