robots.txt(12)

/ Date/ Name
2026.09.23 Planning a safe robots.txt change for a large site

Summary:

A robots.txt change should begin with a precise crawl-control goal, be tested against important paths and be monitored after deployment.

Topics:

Technical SEO 4 min read

Read
2025.06.24 How to test whether a robots directive reaches the right URLs

Summary:

Test robots directives against exact URLs and user-agent groups before changing crawl access.

Topics:

Technical SEO 5 min read

Read
2025.02.28 How AI crawlers use the web, and how to control them

Summary:

What AI crawlers are, why they visit your site, the difference between training and search bots, and how to control them with robots.txt without hurting your search traffic.

Topics:

How the internet works 5 min read

Read
2024.11.22 Web scraping: what is legal and what is fair

Summary:

Scraping sits between legal, allowed and fair. An opinion on where the lines are, from robots.txt to copyright, with questions to ask before you scrape.

Topics:

How the internet works 5 min read

Read
2024.10.16 AI crawlers in your logs: who’s visiting and why

Summary:

Identify automated requests from their user agent and verified network details, then decide which crawlers to allow based on their stated purpose and your site's policy.

Topics:

AI & search 4 min read

Read
2024.05.08 Blocking AI crawlers: GPTBot, Google-Extended and the rest

Summary:

You can tell OpenAI, Google and others not to use your content for AI training with a few lines in robots.txt. What each user agent controls, what blocking does and doesn't do for your search traffic, and how to decide.

Topics:

AI & search 5 min read

Read
2024.03.28 Shopify noindex and meta robots: what’s possible

Summary:

Shopify stores can use meta robots directives for selected pages, but noindex only works when crawlers can access the page. Choose the right control and test the rendered output carefully.

Topics:

Shopify & e-commerce 4 min read

Read
2024.01.29 Robots.txt on Shopify: what you can edit and common mistakes

Summary:

Shopify lets merchants customise robots.txt.liquid, but editing it without a clear crawl problem can block useful pages. Learn what to check, when to edit and how to test changes safely.

Topics:

Shopify & e-commerce 4 min read

Read
2023.12.06 How AI chatbots get their information from the web

Summary:

Where chatbots like ChatGPT get their answers, what browsing changes, and what it means if you run a website.

Topics:

How the internet works 4 min read

Read
2023.11.13 Blocking AI crawlers: should you?

Summary:

Should your site block GPTBot and other AI crawlers? My answer depends on what you publish and why, and here is how to decide without panic.

Topics:

How the internet works 5 min read

Read
2023.11.13 AI crawlers explained: GPTBot, Google-Extended and the rest

Summary:

AI crawlers are bots that read your pages to train or power AI tools. Here is who sends them, how to spot them, and how robots.txt talks to them.

Topics:

How the internet works 5 min read

Read
2023.03.10 Robots.txt and noindex: when to use which

Summary:

Robots.txt and noindex both keep pages out of Google - except they don't do the same thing, and using both on the same page cancels one out. A plain explanation of which to use when.

Topics:

Technical SEO 3 min read

Read