crawlers(4)
2026.02.11 How AI search engines read your website
Summary:
AI search tools fetch, strip down and summarise your pages differently from Google's classic crawler. Here is what they see, and how to make your site easy to read.
Topics:
How the internet works 5 min read
Read2025.03.04 Bots are now a huge share of web traffic: what that means
Summary:
A big slice of web traffic is not human. Here is which bots visit your site, which are useful, which are harmful, and how to stop them skewing your numbers.
Topics:
How the internet works 5 min read
Read2024.12.09 Bad bots vs good bots
Summary:
Not all automated traffic is a threat. Here is how to tell helpful bots from harmful ones, what to allow, and what to block without hurting your site.
Topics:
How the internet works 4 min read
Read2023.12.06 How AI chatbots get their information from the web
Summary:
Where chatbots like ChatGPT get their answers, what browsing changes, and what it means if you run a website.
Topics:
How the internet works 4 min read
Read