robots.txt(12)
2026.09.23 Planning a safe robots.txt change for a large site
Summary:
A robots.txt change should begin with a precise crawl-control goal, be tested against important paths and be monitored after deployment.
Topics:
Technical SEO 4 min read
Read2025.06.24 How to test whether a robots directive reaches the right URLs
Summary:
Test robots directives against exact URLs and user-agent groups before changing crawl access.
Topics:
Technical SEO 5 min read
Read2025.02.28 How AI crawlers use the web, and how to control them
Summary:
What AI crawlers are, why they visit your site, the difference between training and search bots, and how to control them with robots.txt without hurting your search traffic.
Topics:
How the internet works 5 min read
Read2024.11.22 Web scraping: what is legal and what is fair
Summary:
Scraping sits between legal, allowed and fair. An opinion on where the lines are, from robots.txt to copyright, with questions to ask before you scrape.
Topics:
How the internet works 5 min read
Read2024.10.16 AI crawlers in your logs: who’s visiting and why
Summary:
Identify automated requests from their user agent and verified network details, then decide which crawlers to allow based on their stated purpose and your site's policy.
Topics:
AI & search 4 min read
Read2024.05.08 Blocking AI crawlers: GPTBot, Google-Extended and the rest
Summary:
You can tell OpenAI, Google and others not to use your content for AI training with a few lines in robots.txt. What each user agent controls, what blocking does and doesn't do for your search traffic, and how to decide.
Topics:
AI & search 5 min read
Read2024.01.29 Robots.txt on Shopify: what you can edit and common mistakes
Summary:
Shopify lets merchants customise robots.txt.liquid, but editing it without a clear crawl problem can block useful pages. Learn what to check, when to edit and how to test changes safely.
Topics:
Shopify & e-commerce 4 min read
Read2023.12.06 How AI chatbots get their information from the web
Summary:
Where chatbots like ChatGPT get their answers, what browsing changes, and what it means if you run a website.
Topics:
How the internet works 4 min read
Read2023.11.13 Blocking AI crawlers: should you?
Summary:
Should your site block GPTBot and other AI crawlers? My answer depends on what you publish and why, and here is how to decide without panic.
Topics:
How the internet works 5 min read
Read2023.11.13 AI crawlers explained: GPTBot, Google-Extended and the rest
Summary:
AI crawlers are bots that read your pages to train or power AI tools. Here is who sends them, how to spot them, and how robots.txt talks to them.
Topics:
How the internet works 5 min read
Read2023.03.10 Robots.txt and noindex: when to use which
Summary:
Robots.txt and noindex both keep pages out of Google - except they don't do the same thing, and using both on the same page cancels one out. A plain explanation of which to use when.
Topics:
Technical SEO 3 min read
Read