Blog / Build with AI /

Building a broken link checker script with AI

Broken links are the quiet leaks of a website. A page you wrote years ago links to a guide that no longer exists. A product gets removed and ten articles still point at it. Visitors hit an error, trust slips a little, and search engines notice a site that isn’t looked after.

You can use a plugin or an online crawler. You can also build a small checker yourself, in an afternoon, with AI doing the typing. The version below is deliberately modest: it checks the links on one page. That’s enough to learn how it works, and enough to be useful.

The finished thing in one line: a script that takes a page address, collects its links, requests each one, and lists the ones that fail.

What “broken” means

A link isn’t just “works” or “doesn’t”. The server replies with a status code:

  • 200 range: fine.
  • 301 / 302 range: redirected. It works but may be sending people on a detour.
  • 404: not found. Broken.
  • 410: gone on purpose.
  • 500 range: the server has a problem, maybe temporary.

If these are new, HTTP status codes explained: 200, 301, 404, 500 is the primer. Your script’s job is to report anything that isn’t a clean 200, and tell you which kind.

Step 1: the plan

Write the plan in plain steps before prompting:

  1. Fetch the page’s HTML.
  2. Find every <a href="...">.
  3. Turn relative links (/about/) into full addresses.
  4. Skip links that aren’t web pages: mailto:, tel:, # anchors, javascript:.
  5. Remove duplicates.
  6. Request each link and record the status.
  7. Print the problems.

Step 2: the prompt

Write a Node.js script (Node 18 or later, no external packages) that takes a page URL as a command-line argument. Fetch the page, extract all href values from anchor tags, resolve relative URLs against the page URL, skip mailto, tel, javascript and fragment-only links, and remove duplicates. Check each link with a HEAD request, falling back to GET if HEAD fails or returns 405. Use a 10 second timeout. Check at most 5 links at a time so we are polite to servers. Print a table of links whose final status is not 200, with the status and whether it was redirected. Add comments.

Each constraint solves a known problem. HEAD requests are lighter than GET, but some servers reject them. A timeout stops one slow site hanging the whole run. A limit of five at a time keeps you from flooding a server.

Step 3: a shape worth recognising

The heart of the script is a loop that handles failures without crashing:

async function check(url) {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), 10000);
  try {
    let res = await fetch(url, { method: 'HEAD', redirect: 'manual', signal: controller.signal });
    if (res.status === 405 || res.status === 403) {
      res = await fetch(url, { method: 'GET', redirect: 'manual', signal: controller.signal });
    }
    return { url, status: res.status };
  } catch (err) {
    return { url, status: 'error', note: err.name === 'AbortError' ? 'timeout' : err.message };
  } finally {
    clearTimeout(timer);
  }
}

With redirect: 'manual', a redirect shows up as a 301 or 302 rather than being followed silently, so you can report it. The try, catch and finally block means that a dead domain gives you a note in the results instead of killing the script.

You’ll want the AI to explain the concurrency part (checking five at a time). It’s the fiddliest bit, and it’s good to understand before you run the script against a large page. If you want to see why failing gracefully matters, error handling: making tools fail gracefully is a good companion.

Create a test page with a few links on purpose:

  • A good link.
  • A page on your own site that you know returns 404.
  • A link that redirects (try the http version of a site that forces https).
  • A domain that doesn’t exist.
  • A mailto: link, to check it’s skipped.
  • The same link twice, to check it’s counted once.

Run the script and compare the output against what you know. Never trust a checker you haven’t tested on known failures.

False alarms you’ll meet

  • Sites that block scripts and return 403 or 429 to everything that isn’t a browser. A link flagged this way might be fine for people. Open it by hand before changing anything.
  • Temporary 500 errors. Check again later before declaring a link dead.
  • Soft 404s. A page that says “not found” but returns 200. A status-code checker can’t see these.
  • Links built by JavaScript. A plain fetch sees only the HTML sent by the server. Links added later in the browser won’t be in the list.
  • Redirect chains. Several hops in a row are worth fixing, even if each one “works”.

Be a polite visitor

Checking your own site is fine. For external links, keep the pace low, never loop forever, and don’t hammer one host with dozens of requests at once. The limit of five concurrent requests above is there for a reason. If a site asks not to be crawled, respect that.

Fixing what you find

For each broken link, decide:

  1. Update it to the new address, if the page moved.
  2. Replace it with a better source, if the page is gone.
  3. Remove the link but keep the sentence, if nothing replaces it.
  4. Redirect it, if the broken link is on your own site and other pages (or other sites) point at the old URL.

If a lot of internal links break at once, something bigger has happened, like a changed URL structure. That’s a job for redirects, not link-by-link repair.

Where to take it next

  1. Crawl the whole site by following internal links, with a page limit.
  2. Read links from your sitemap instead of a single page.
  3. Save results to a CSV so you can compare week to week.
  4. Run on a schedule and email only when something new breaks.

Add them one at a time, as discussed in prompt chaining: building in steps.

Key takeaways

  • Broken-link checking is just fetching URLs and reading status codes.
  • Skip non-web links and deduplicate before you start.
  • Use timeouts, limits and fallback requests.
  • Test on known good and known bad links.
  • Check a flagged link by hand before acting.

FAQ

Should I use a plugin instead?

If you run WordPress and want ongoing monitoring, maybe. Building your own is great for understanding, and for one-off audits.

How often should I check?

For a small site, monthly is a sensible rhythm. After a redesign or migration, check straight away.

A few won’t sink a site, but they waste visitors’ time and they reflect poorly on upkeep. Fix the ones on important pages first.

Run it on your most-visited page. How many links failed, and were they real failures?