Blog / Shopify & e-commerce /

Measuring the effect of a catalogue change without claiming causation

A category page improves after a catalogue change, and the chart goes up. It’s tempting to say the change caused it, especially after the team worked hard to make it happen. But search visibility also moves with seasonality, stock, competitors, campaigns, site releases and Google’s own systems.

A careful review can show what happened without pretending to know more than the evidence supports. The goal is to measure a change in a way that informs the next decision. That takes a defined intervention, a baseline and honest language about uncertainty.

Decide what changed

Write down the catalogue change precisely. Did product names, attributes, collection membership, availability or URL structure change? Record when the update reached the live site and which pages it touched. If several things changed together, you can describe the combined outcome, but you can’t neatly isolate one part.

Choose a measure connected to the intended outcome. If the goal was to make products easier to find, look at discovery and relevant search visibility. If it was clearer product selection, consider engagement with key information or support questions. Don’t pick a metric because it moved in a pleasing direction.

Establish the comparison

Use a period before the change and one after that make sense for the page and the business. Think about weekday patterns, seasonality, campaign timing, stock and the time crawlers need to process changes. A comparison with a different seasonal period can mislead even when the date ranges are the same length.

Where you can, select comparable pages that didn’t get the change. A comparison group gives context, but it’s rarely a perfect control, because pages differ in demand, product, competition and internal links. State those limits instead of presenting the group as proof.

Preserve the raw evidence

Record the source, filters and export date. In Search Console, check the exact property, search type, country, device and page or query filters. In analytics, confirm event definitions and consent configuration. For catalogue data, keep the version or export you used. That lets someone reproduce the analysis instead of trusting a chart pasted into a slide.

Segment where it answers a question, not because segmentation is possible. Too many small segments invite stories built from random fluctuation. Keep the main measure visible and label secondary observations as exploratory.

Watch for data quality traps

Before comparing periods, check product identifiers and URLs stay consistent. A handle change can split reporting across two pages, and a catalogue export may assign a new identifier to an existing product. Decide whether the analysis is page-level, product-level or collection-level, and align the records. Document the mapping so the same product isn’t counted twice or left out.

Consider search query changes as well. A catalogue edit may improve wording and make the page appear for a different set of searches. Aggregate clicks can stay flat while the mix of intent improves, or rise because a broader query entered the report. Inspect query and landing-page patterns before concluding the change did nothing.

Separate observation from explanation

Say what the data shows: impressions rose for a defined page group after a specific change, or clicks stayed similar while availability changed. Then list plausible explanations, including the catalogue work and external factors. Use wording like “the change coincided with” unless the design supports a stronger claim.

Don’t turn correlation into a success claim. A rise may reflect seasonal demand, a campaign or a broader search change. A decline may follow stock problems or an unrelated release. Search data helps direct an investigation, and it rarely explains every cause alone.

Check the implementation too

Before interpreting performance, confirm the change was deployed as intended. Did the right products move into the collection? Are URLs and canonicals correct? Are titles and descriptions rendered? Did internal links update? Don’t assume the task shipped correctly because a ticket says complete.

If the implementation was partial, note the affected pages and restart the review window if needed. Don’t quietly drop inconvenient URLs from the analysis. A transparent record beats a polished chart with hidden exclusions.

Make a decision from the result

The outcome may support scaling the change, revising it, gathering more evidence or leaving it in place while you monitor. Each is a legitimate decision. Include your confidence and what would change your mind. Don’t use one result to justify a site-wide rollout when the test involved a small or unusual set of products.

Use a short report: question, intervention, pages, period, measures, observed outcome, confounders and next action. A short limitations note tells decision-makers how much weight to give the result. I’d rather report an honest “inconclusive, here’s what we’ll check next” than claim credit I can’t support.

If a result is uncertain, keep the next step modest. Keep the change, test it on another comparable group, fix an implementation issue or wait for a better window, and say when you’ll review again.

When you review your next catalogue change, can you tell the evidence apart from the explanation you hope is true?