Blog / How the internet works /

Bad bots vs good bots

Open your server logs for the first time and you’ll get a shock. A big share of the requests aren’t people. They’re programs.

Some of those programs are the reason you show up in search results. Others are trying to guess your login, copy your content or fill your forms with junk. Telling them apart is a skill every site owner needs.

What a bot is

A bot is software that visits web pages automatically. That’s all. “Bot” is neither good nor bad; it’s a method. What matters is who sent it and what it does when it arrives.

The good bots

These are the visitors you want.

  • Search engine crawlers. Googlebot, Bingbot and others. Without them, you don’t get indexed.
  • Social preview fetchers. When you paste a link into a chat or social post and a preview card appears, a bot fetched the title and image.
  • Uptime monitors. The tools that check your site is alive (more on those in the uptime monitoring setup I use for client sites).
  • Feed readers and tools you’ve signed up for. SEO crawlers you run yourself, analytics checks and payment webhooks.
  • Accessibility and archive services. Some libraries and archives crawl openly and politely.

A good bot normally identifies itself honestly, follows your robots.txt rules and slows down if you ask it to. If you want to know what robots.txt does, there’s a quick read in robots.txt explained in five minutes.

The bad bots

  • Credential stuffing and brute force. Bots trying stolen or guessed passwords against /wp-login.php or your shop’s account page.
  • Scrapers. Programs copying your product descriptions, prices or articles, sometimes to republish them.
  • Spam bots. They fill comment boxes and contact forms with links and junk.
  • Vulnerability scanners. They probe for outdated plugins and known holes.
  • Fake traffic. Bots that inflate visits or fake referrals and dirty up your analytics.
  • Card-testing bots. On shops, scripts that try stolen card numbers through checkout, which costs real money in fees and chargebacks.

The grey area: AI crawlers

AI companies run crawlers to collect training data and to fetch pages for answers. Some are well-behaved and publish how to block them. Whether they are “good” or “bad” depends on what you want. I went through the main AI crawlers and how to control them and what shows up when you look for them in your logs.

Blocking them is your decision. The point is that it should be a decision, not an accident.

How to tell which is which

Names can lie. Any bad bot can call itself “Googlebot” in its user agent string. A few checks help.

  1. Look at the behaviour. A real crawler requests pages at a sane pace. A scanner hits hundreds of odd addresses like /xmlrpc.php or /.env in seconds.
  2. Verify the claim. Search engines document how to confirm their crawler is genuine, for example by checking the visiting address through a reverse DNS lookup. If a “Googlebot” visits from a random hosting company, it’s a fake.
  3. Check what it asks for. Real crawlers fetch pages, images and sitemaps. Bad ones hammer login URLs, search pages and checkout.
  4. Watch for odd patterns. Hundreds of visits, one second each, zero scroll or engagement, all from the same country or network.

What to do about bad bots

Start with the cheap, safe steps. You don’t need to build a fortress.

Keep the site updated. Most scanners hope for old plugins. My list of website security basics starts here for a reason.

Limit login attempts and add two-factor authentication for admin accounts.

Use a firewall or CDN with bot protection. Cloudflare’s free plan offers basic bot and rate-limiting settings, and I covered the practical ones in Cloudflare security settings for small sites.

Add a spam filter or challenge to forms. I’ll go through the options for contact forms in a separate post.

Rate-limit expensive URLs such as search, login and checkout.

Use robots.txt for polite bots only. It’s a request, not a lock. A malicious bot will ignore it. Don’t hide private pages with it: it’s a public file, and listing paths can point attackers at them.

What not to do

  • Don’t block by country on a whim. You may cut off real customers and travellers.
  • Don’t block every unknown user agent. You will probably catch a legitimate service.
  • Don’t challenge search crawlers. A CAPTCHA wall in front of Googlebot can get pages dropped.
  • Don’t trust analytics alone. Many bots never run JavaScript, so they won’t show up in your reports, only in server logs.
  • Don’t panic at the number. Some bot traffic is normal for any public site.

Why site owners should care

Bad bots cost you in ordinary, boring ways: slower pages for real visitors when your server is busy, higher hosting bills on metered plans, polluted analytics that mislead your decisions, and now and then a real security incident. Good bots are the lifeblood of being found. Blocking everything and allowing everything both fail.

A simple routine

Once a month, take ten minutes:

  1. Look at the top user agents and top requested URLs in your host or CDN dashboard.
  2. Note anything strange, especially login and checkout hits.
  3. Confirm the real search crawlers are still getting through.
  4. Adjust one rule if something clearly needs it.

Key takeaways

  • A bot is a method, not a verdict. Judge by who sends it and what it does.
  • Good bots (search, previews, monitoring) need to get in; bad bots (brute force, scraping, spam) don’t.
  • User agent names can be faked; behaviour and verification are more reliable.
  • Layer cheap defences: updates, login limits, a CDN firewall, form protection.

When did you last look at what is actually visiting your site?