crawlers(4)

/ Date/ Name
2026.02.11 How AI search engines read your website

Summary:

AI search tools fetch, strip down and summarise your pages differently from Google's classic crawler. Here is what they see, and how to make your site easy to read.

Topics:

How the internet works 5 min read

Read
2025.03.04 Bots are now a huge share of web traffic: what that means

Summary:

A big slice of web traffic is not human. Here is which bots visit your site, which are useful, which are harmful, and how to stop them skewing your numbers.

Topics:

How the internet works 5 min read

Read
2024.12.09 Bad bots vs good bots

Summary:

Not all automated traffic is a threat. Here is how to tell helpful bots from harmful ones, what to allow, and what to block without hurting your site.

Topics:

How the internet works 4 min read

Read
2023.12.06 How AI chatbots get their information from the web

Summary:

Where chatbots like ChatGPT get their answers, what browsing changes, and what it means if you run a website.

Topics:

How the internet works 4 min read

Read