gptbot(4)

/ Date/ Name
2025.02.28 How AI crawlers use the web, and how to control them

Summary:

What AI crawlers are, why they visit your site, the difference between training and search bots, and how to control them with robots.txt without hurting your search traffic.

Topics:

How the internet works 5 min read

Read
2024.05.08 Blocking AI crawlers: GPTBot, Google-Extended and the rest

Summary:

You can tell OpenAI, Google and others not to use your content for AI training with a few lines in robots.txt. What each user agent controls, what blocking does and doesn't do for your search traffic, and how to decide.

Topics:

AI & search 5 min read

Read
2023.11.13 Blocking AI crawlers: should you?

Summary:

Should your site block GPTBot and other AI crawlers? My answer depends on what you publish and why, and here is how to decide without panic.

Topics:

How the internet works 5 min read

Read
2023.11.13 AI crawlers explained: GPTBot, Google-Extended and the rest

Summary:

AI crawlers are bots that read your pages to train or power AI tools. Here is who sends them, how to spot them, and how robots.txt talks to them.

Topics:

How the internet works 5 min read

Read