Blog / AI & search /

AI crawlers in your logs: who’s visiting and why

There’s a strange new bot name in your server logs. Is it training an AI model? Powering a search answer? Probing for weak spots?

The user-agent string gives you a clue, but it’s not proof of who is visiting or why.

AI-related crawlers fetch public pages for different reasons. Some are linked to model training. Others fetch a page because a person asked a question. I’d start with the actual requests and the provider’s current documentation, then set a clear policy on whether to allow or disallow each one.

Read the logs before changing robots.txt

Look at the user-agent, requested URLs, response status, timestamps, frequency and source network information available to you. Identify whether requests are concentrated on public content, query parameters, login pages or endpoints that should not be crawled.

A user-agent is just a string the client sends. It can be copied or spoofed, so the name alone proves nothing. If the provider publishes IP ranges or verification guidance, compare the source against it. If the traffic is unusual or heavy, I’d ask the host or security team for help.

Separate ordinary crawling from harmful behaviour. A bot hammering a low-value parameter URL may need rate limits whether or not it’s labelled “AI”. Not every AI user agent is a security incident.

Understand what each name means

In October 2024, GPTBot was a documented OpenAI crawler associated with content that could be used to improve its generative AI models. OpenAI also documented ChatGPT-User for some user-initiated page visits. Those uses are different: one is an automated crawler, while the other can be triggered by a person asking for a page to be fetched.

Google-Extended is a robots.txt control token, not a separate user-agent string. Google says it manages certain uses of content for its generative AI products, and it doesn’t affect inclusion in Google Search or act as a ranking signal. So you won’t find a Google-Extended bot in your access logs.

Crawler names and purposes can change. Check the vendor’s documentation for the date and user agent you actually observe. Do not assume that a new name has the same policy as an older one.

Decide what access you want to allow

Robots.txt expresses preferences to cooperative crawlers. It isn’t authentication, and a crawler that ignores it can still fetch public content. Protect private information with access controls, not disallow rules.

Write a policy that fits the site’s goals. You might want search crawlers on public pages while limiting a training crawler, or allow user-triggered fetches because they help people find you. I don’t think there’s one right answer. It depends on your licensing, privacy, security and business priorities.

Use the correct token exactly as the provider documents it. A typo or overly broad rule can affect a different crawler than intended. Test robots.txt syntax and review the file after deployment. Keep a record of why the rule exists so a future editor does not remove it by accident.

Think about the content boundary as well as the technical one. Public marketing pages, private customer data and admin areas shouldn’t be treated alike. If your organisation has a policy on machine use of its content, decide with the business or legal owner and record the date. A robots.txt rule states a preference. It doesn’t create a licence, guarantee compliance or stop a determined party from reading a public page.

Balance access with server health

Check request rate and response codes. If a crawler sends a high volume of requests, coordinate with the host about rate limiting, caching or a direct request policy. Do not block all bots at the firewall without checking whether search and monitoring systems also use that route.

Use logs to find whether the crawler is requesting duplicate URLs, filtered pages or resources that are expensive to generate. Canonicalise or limit URL patterns where appropriate, and protect administrative endpoints separately. A robots rule does not stop a request from reaching the server in all circumstances.

If the traffic threatens availability, use the same incident process you would for any automated load: gather evidence, set a measured control and monitor the effect. A bot label is not enough to decide what the risk is.

Keep expectations realistic

Allowing a crawler does not guarantee that your content will be cited, indexed or used in a particular answer. Blocking a crawler does not remove content already copied or retrieved by another route. Review the provider’s stated policy and decide what your own site can control.

If your organisation needs a legal or licensing decision, involve the appropriate person. The technical team can implement and verify the crawler rules, but should not invent a content licence or promise a particular AI outcome.

For general AI and SEO context, see AI-written content and Google: what Google has actually said and AI content detectors: why I don’t rely on them. For crawler controls, see Robots.txt on Shopify: what you can edit and common mistakes.

Review policy when the business changes

A crawler policy may need to change when the site publishes new types of content, adopts a licensing policy or begins using a service that depends on fetching public pages. Assign an owner to review the rules and keep the reason beside each one. Without that explanation, a future developer may remove a restriction because it looks accidental or extend it without understanding its effect.

Check the provider documentation again when you see a new user agent or a meaningful change in request behaviour. Do not assume the bot name, claimed identity or purpose is permanent. If you block a crawler, monitor whether the rule affects the audience or tools you intended to serve. If you allow one, continue protecting private routes through authentication and access controls.

Take one crawler entry from your logs and check it against the provider’s documentation. What is it actually requesting, and what policy do you want to apply?