Blog / Case studies /
Planning an SEO test when you cannot control every variable
You update a group of pages to improve their internal links. Meanwhile stock changes, another team publishes new content and search demand shifts. The report moves, but the test can’t tell you for certain which change caused it.
SEO work happens on live sites where you can’t control every variable. That doesn’t make testing pointless. It means you design the question, comparison, timeline and limitations carefully, so the team learns something without claiming more than the evidence supports.
Decide what the test can answer
Start with a specific hypothesis. For example: “Adding relevant links from these category pages may help visitors and crawlers reach a selected group of products.” That beats “improve SEO”. State which pages will change, what you expect to see and what evidence would make you reconsider.
Separate implementation from outcome. You can verify the links were added to the intended templates. You can observe whether crawl paths or search metrics change afterwards. The first is a direct check; the second is affected by many other factors. I don’t treat a finished implementation as proof of ranking impact.
Choose a primary measure that matches the question, plus a few supporting checks. If the test is about discoverability, look at crawlability, internal link paths, indexation and relevant search exposure. If it’s about a user journey, check clicks and the next action. Don’t track a dozen metrics without deciding what each one tells you.
Choose a comparison that is credible
When possible, select a comparable group of pages that will not receive the change during the test window. Match them on page type, prior performance, seasonality and business role where practical. Avoid comparing unrelated page groups and interpreting the difference as an experiment result.
Sometimes there’s no suitable control group. Then compare the changed pages with their own prior period, check the broader site trend and record known confounders. A staged rollout may help, if there’s enough stable data and the team understands the method. I’d rather admit a weak comparison than decorate it with statistical language.
Preserve the page list and baseline before making the change. Record the URL, template, market, date range, filters and any known limitations. If pages are added or removed during the test, note the change and consider whether the group remains comparable.
Record the surrounding changes
Keep a change log for releases, campaigns, promotions, inventory, migrations, content updates and measurement changes during the test. A relevant event does not automatically invalidate the work, but it affects interpretation. If a site redesign happens at the same time, the test may no longer isolate the internal-link change.
Ask the teams that own the affected pages to tell you when a meaningful change is planned. You don’t need a site freeze. You just need the timeline visible. If an operational change can’t be delayed, record it and decide whether the test continues, pauses or gets reframed.
Use the same data sources and definitions throughout. Search Console and analytics reports can change as data arrives or filters are adjusted. GA4 key events need to be configured consistently. Capture the extraction date and note any sampling, reporting lag or tracking changes.
Avoid changing the treatment mid-test
Define what “treated” means. If the test is adding links, specify the placement, anchor approach, number or type of pages and any exceptions. Otherwise each page may receive a different intervention and the team will not know what it evaluated.
Keep the treatment stable for a pre-agreed observation period unless a problem forces a change. Don’t keep editing titles, content and links because early data looks disappointing. If you must change something, record it and decide whether it starts a new test phase.
Avoid picking only the successful examples afterwards. Report the full group and explain exclusions. If some pages couldn’t be updated, show that and say how it affects interpretation. A test that quietly drops poor performers isn’t fair.
Interpret results with restraint
Look at direction, size, consistency and context. A few pages may improve while others remain flat. The average may conceal important differences. Segment only where the sample and question support it; too many cuts make patterns appear stronger than they are.
If there’s no clear movement, the honest conclusion may be that the result is inconclusive, or that the change had no detectable effect in the period observed. That isn’t proof the intervention never works. State the time window and data limits.
I don’t claim causality from a before-and-after chart alone. Search results change, competitors publish, products go in and out of stock and demand shifts. Explain which alternatives you considered and what extra evidence would strengthen the conclusion.
Turn learning into a decision
A test should lead to a next step: keep the change, revise it, expand carefully, run a better test or stop investing in the idea. Write down why and what the team learned. If the result changes a recommendation, link it to the original hypothesis and evidence.
Share the findings with page owners in terms they can use. Include the affected pages, implementation status, observed metrics, confounders, confidence and next action. Avoid presenting an SEO test as a guarantee of future results.
Test honestly in a changing site
You may not control every variable, but you can control how clearly you define the question and document what happened. A careful test provides bounded evidence rather than a sweeping claim.
Related reading
- Building a small SEO test around one page template
- What a careful ecommerce SEO experiment can and cannot prove
- AI and SEO experiments: separating a useful test from a demo
Before your next SEO experiment, list the changes that could affect the result and decide how you will record them. What conclusion will still be valid if one of those variables moves during the test?