Blog / Technical SEO /
Diagnosing unexpected indexing after a CMS migration
The migration is live. The new design looks great. Everyone’s relieved.
Then Search Console starts filling up with URLs nobody expected. Old pages that should have gone. New CMS routes that were never meant to be public.
The temptation is to start blocking things in robots.txt straight away. Don’t. First work out which system created each URL and what the site is actually telling search engines about it.
Build a list of what changed
Start with a pre-migration URL list if one exists. Compare it with the current sitemap, a crawl of the live site, Search Console pages and analytics landing pages. Each source is incomplete in a different way: a sitemap reflects what the site lists, a crawl reflects what a crawler can reach, and search reports include URLs discovered over time. Combine them to identify patterns rather than treating any single export as the full truth.
Group unexpected URLs by type. Look for old paths, duplicate versions, query parameters, staging routes, CMS preview links, tag archives or pages with inconsistent trailing slashes. Sample several from each group and record the response code, final destination, canonical, robots directives and page content. A URL in a report is a clue, not proof of a current problem. When I investigate one of these on a client site, I sample before I touch anything.
Trace the source
Check the migration map, CMS configuration, redirect rules, internal links and sitemap generation. A route may be created by a taxonomy, filter, app or theme template. Old internal links can keep a retired URL discoverable after a migration, and a redirect may send it through more than one destination. The current page can also carry a canonical pointing somewhere unexpected.
Ask what the URL is supposed to do. Is it a useful page that should be indexed, a duplicate that should consolidate with a preferred page, an old page with a close replacement, or a route that should no longer exist? The right technical response depends on that answer. Don’t hide an unmade decision behind a blanket noindex. I see that shortcut a lot, and I don’t trust it.
Correct the signal at its source
For useful pages, give them stable internal links, appropriate metadata and inclusion in the sitemap where intended. For duplicates, choose a preferred version and make links and canonicals consistent. If a page has moved, redirect it to a genuinely equivalent destination. If there is no equivalent and the page should be gone, return the appropriate not-found or gone response rather than sending every old URL to the home page.
Be careful with robots.txt. Blocking a URL can prevent crawlers from seeing a noindex directive on the page. Robots directives, canonicals, redirects and sitemap entries should tell a coherent story. Remove staging access or accidental publication at the source, then inspect the live response after deployment.
Verify in stages
Test a small set of representative URLs in staging where possible. After deployment, check the response and final page, crawl the important sections, and confirm that internal links point to the preferred destinations. Only request reprocessing once the page itself is right. A request won’t repair a bad route.
Keep a migration issue log with the URL pattern, decision, owner and verification date. That record helps distinguish a genuinely new problem from old URLs still being processed. Which unexpected URL pattern can you trace back to a specific migration rule or CMS setting today?
Watch for migration artefacts
Unexpected indexing can come from temporary migration choices that were never removed. A staging site may have been crawlable, a development hostname may still resolve, or a preview route may have entered a sitemap. Check whether the hostnames belong to the live property and whether the content is accessible without authentication. If a non-production copy is public, restrict it at the source and ensure it cannot compete with the canonical site.
Also inspect language, category and pagination patterns. A CMS can create archive pages automatically, while imported content may carry old slugs or duplicated metadata. A template with an empty field can produce near-identical titles across many pages. Sample the page types and inspect how the migration mapped each field; fixing the template may be better than editing dozens of URLs individually.
If Google continues to show an old URL after a correct migration, check its live response and whether any internal or external links still point to it. Search systems can take time to reflect a change. Repeatedly changing directives while waiting can create conflicting signals. Keep a list of examples and recheck them after the site has been stable for a reasonable period.
Coordinate technical and editorial owners
Migration issues often cross roles. A developer may control redirects, an editor owns page content and a marketer understands which URLs have value. Bring the right people together before retiring a large URL group. Share a sample and describe the user impact, not just the crawl error. A decision log can record what was retained, consolidated or removed and who approved it.
For high-impact sections, test a small batch first. Verify the destination and the internal links, then expand the change once the result is understood. This limits the risk of a broad rule catching pages that should remain available. Make sure the sitemap reflects the intended final set and that canonical tags agree with the chosen URL.
Keep the migration map after launch. I keep mine for every client migration, because it’s the first thing I reach for in later audits. If the unexpected pages are growing rather than shrinking, look for a route generator or internal link that keeps publishing them.
Related reading
- Fixing ‘Crawled – currently not indexed’
- Programmatic SEO: when it works and when it’s spam
- Fixing ‘Discovered – currently not indexed’
So, your next step: pick one migration rule and inspect it at its source. It’s far quicker than treating each URL as a separate error.