google-extended(1)
/ Date/ Name
2024.05.08 Blocking AI crawlers: GPTBot, Google-Extended and the rest
Summary:
You can tell OpenAI, Google and others not to use your content for AI training with a few lines in robots.txt. What each user agent controls, what blocking does and doesn't do for your search traffic, and how to decide.
Topics:
AI & search 5 min read
Read