Blog / Technical SEO /

Technical SEO for a site with thousands of pages

The crawl finishes. Thousands of URLs, a wall of warnings, and no idea where to start. Sound familiar?

On a large site, technical SEO isn’t about clearing every row in a spreadsheet. It’s about understanding how URLs are generated, which pages matter, where a pattern begins and which change helps without breaking another section.

First, make the site legible. Group URLs by template, purpose, indexability and business role. Then connect the crawl to Search Console, analytics and whoever owns the CMS or product data. What looks like a page problem is often a template problem, and a strange crawl result may be a deliberate filter or internal tool. Context turns a pile of URLs into a plan.

Key takeaways

  • Build a URL inventory that explains what pages exist and why.
  • Segment before crawling, and use representative samples to diagnose patterns.
  • Prioritise indexable pages that serve a real search or business need.
  • Fix the templates, rules or data sources behind repeated issues.
  • Test changes on a limited set, document them and monitor the result.

Start with an inventory

Large sites pile up URLs from products, categories, filters, search results, pagination, campaign parameters, archives, regional versions and old migrations. A crawler finds many of them, but that doesn’t tell you which ones should exist in search.

Create a taxonomy of page types. For each type, note its purpose, source system, owner, expected indexability, canonical pattern and how it is linked. The inventory can be a spreadsheet, a database table or documentation in the team’s normal system. It needs to be maintainable, not impressive.

Compare what the site means to publish with what search systems see, using Search Console’s indexing reports, sitemaps, server logs if you have them, analytics landing pages and a crawl. Each source has limits. Search Console doesn’t show every URL, analytics only counts measured visits, and a crawl sees what its settings allow. I use them together to find the gaps.

Segment the crawl

One crawl of every possible URL creates noise, especially when filters and parameters multiply. Set limits and include or exclude sections on purpose. Crawl a representative slice of each template, then widen the net where the evidence points to a problem. Keep your crawl settings so someone else can reproduce the work.

Group findings by pattern: missing or duplicate titles, error responses, redirect chains, canonicals pointing elsewhere, orphaned pages, blocked resources, thin templates or unexpected parameter URLs. Count affected URLs only after verifying the rule that identifies them. A tool may flag a pattern that is acceptable for one template and harmful for another.

Separate warnings from problems. A missing meta description on a low-value filter URL isn’t as urgent as an indexable product template returning errors. Prioritise by audience impact, search visibility, business importance, risk and effort, not by URL count alone.

Decide what should be indexed

Indexation is a selection problem. A big site has plenty of valid pages with little unique value: internal search results, empty categories, near-duplicate filters, expired products, parameter variants. Ask what the page does for a user before deciding whether it should be discoverable.

For every page type, define the intended canonical, robots behaviour and sitemap inclusion. Keep these signals consistent. A page marked noindex should not be treated as an important search landing page; a sitemap should not be filled with URLs the business does not want indexed. Canonical tags are hints, so they cannot repair inconsistent internal links or materially different duplicates.

Faceted navigation needs particular care. Filters can make a catalogue usable, but combinations can generate many URLs. Decide which facets represent meaningful search destinations and which are browsing states. Use the platform’s controls carefully, and test whether crawlers can reach the intended pages without being trapped in endless combinations.

Fix causes, not rows

When many pages share an issue, find the template, plugin, import rule or CMS field behind it. Fixing the source lasts longer than editing URLs by hand. Before changing a template, list the page types it touches and test a sample of each. One shared component can fix many pages, or break them all at once.

For example, a title-generation rule may omit an important product attribute. The durable fix may be a data-quality process or template adjustment, not rewriting titles one by one. A redirect chain may come from legacy rules layered over time; update the source links and map old URLs to final destinations rather than adding another redirect on top.

Work with developers using clear examples: affected URLs, expected behaviour, evidence, priority and acceptance criteria. “Fix SEO” is not a buildable ticket. Explain what the page should return, which signals should be present and how the change can be tested.

Monitor the important journeys

Technical SEO is not only about crawlers. Test the paths users take: search result to product, category to filter, article to service, form submission to confirmation. Check mobile navigation and core templates. A site can return a successful status code while a customer cannot complete an important action.

Use monitoring for uptime and key workflows where possible. Keep a change log of deployments, migrations, robots edits and template changes. If Search Console reports a shift, you can compare it with known releases instead of guessing from memory.

Make large changes safely

For migrations, platform changes or indexation rules, stage the work and test a small but representative set before broad rollout. Keep backups and rollback instructions. Validate redirects, canonicals, sitemaps, structured data and analytics after deployment. Crawl the live site again to confirm that the intended rules took effect.

Avoid changing unrelated systems at once. If you alter navigation, templates, internal links and robots directives in one release, diagnosing an unexpected result becomes harder. Break work into reviewable changes and record why each one was made.

A practical review cadence

A large site will continue to change as products, content and systems change. Decide how often to crawl each section based on how frequently it changes and how important it is. Review high-risk templates after releases and run broader audits periodically. Assign owners for product data, redirects, technical configuration and editorial pages.

The goal is not a report with zero warnings. It is a site whose important pages can be found, understood and used, with enough governance to keep it that way. What would you learn first from your URL inventory?