gptbot(4)
2025.02.28 How AI crawlers use the web, and how to control them
Summary:
What AI crawlers are, why they visit your site, the difference between training and search bots, and how to control them with robots.txt without hurting your search traffic.
Topics:
How the internet works 5 min read
Read2024.05.08 Blocking AI crawlers: GPTBot, Google-Extended and the rest
Summary:
You can tell OpenAI, Google and others not to use your content for AI training with a few lines in robots.txt. What each user agent controls, what blocking does and doesn't do for your search traffic, and how to decide.
Topics:
AI & search 5 min read
Read2023.11.13 Blocking AI crawlers: should you?
Summary:
Should your site block GPTBot and other AI crawlers? My answer depends on what you publish and why, and here is how to decide without panic.
Topics:
How the internet works 5 min read
Read2023.11.13 AI crawlers explained: GPTBot, Google-Extended and the rest
Summary:
AI crawlers are bots that read your pages to train or power AI tools. Here is who sends them, how to spot them, and how robots.txt talks to them.
Topics:
How the internet works 5 min read
Read