google-extended(1)

/ Date/ Name
2024.05.08 Blocking AI crawlers: GPTBot, Google-Extended and the rest

Summary:

You can tell OpenAI, Google and others not to use your content for AI training with a few lines in robots.txt. What each user agent controls, what blocking does and doesn't do for your search traffic, and how to decide.

Topics:

AI & search 5 min read

Read