Blog / Shopify & e-commerce /

A decision log for ecommerce experiments

The store test ended a while ago. The new layout is still live, but nobody can remember whether it won, whether tracking failed or whether the designer just liked it better. Experiments turn into permanent changes this way, by accident and without a decision.

A decision log stops that. I want it to say what the team believed, what it observed and what it decided. It should be short enough to keep up while the work is happening, with enough evidence that someone joining later can follow the reasoning without finding the original meeting.

Write down the decision the experiment serves

Start with the business question, not the tool. “Should we make delivery information easier to see on product pages?” gives the team a decision. “Try a new accordion” describes an implementation without explaining why the change deserves attention.

Record the problem and the evidence available before the test. That might be repeated delivery questions, an observation from page review or a confusing layout. Label assumptions as assumptions, and don’t rewrite the original uncertainty once the result is in.

I’d always name the person who will make the final decision. They need to understand the commercial trade-off and the limitations of the evidence. An experiment with no decision owner can generate plenty of discussion and still leave the store in an ambiguous state.

Describe exactly what changed

Keep the affected URLs or product identifiers, the original state and the proposed change. Save screenshots where presentation matters, but include text describing the difference. Images alone can hide details about when a feature appears or which visitors receive it.

Note the start date, any interruptions and the point when the change actually reached visitors. The deployment time and the first valid measurement period may differ. If something was corrected mid-run, record it instead of pretending nothing changed.

Include related activity that could affect interpretation: a promotion, stock disruption, a changed traffic source or another release on the same pages. You do not need to catalogue every event in the business. Capture the context that could plausibly explain the outcome.

Agree the evidence before looking at the result

State what the team will measure and why it connects to the question. A buying decision may involve purchases, while a clarity improvement may also need support feedback or observed task completion. Choose a primary measure and a small number of checks that protect against an undesirable side effect.

Write down how the data is collected and what is missing. If analytics cannot reliably distinguish the affected pages, fix that before leaning heavily on its numbers. If you use GA4, name relevant important actions as key events and describe what each event actually records.

Set a review condition suited to the method. A controlled test needs a proper design and analysis plan, while a before-and-after observation makes weaker causal claims. Calling both an “experiment” doesn’t make them equally strong.

Separate observation from interpretation

When reviewing the outcome, first record what happened. Then explain what the team thinks it means. “Fewer questions were tagged delivery” is an observation; “the new layout removed confusion” is an interpretation that may need more support.

Write down alternative explanations worth considering. Perhaps fewer orders meant fewer opportunities for questions. Perhaps a colleague changed how tickets were labelled. That’s not a reason to dismiss every finding. It just stops a convenient story replacing the analysis.

If the result is inconclusive, say so. The team can still decide that a change is clearer, cheaper to maintain or worth another test, provided the reason is honest. A practical decision does not require inventing a statistical victory.

Record the action and its conditions

Choose what happens next: keep the change, revert it, revise it or investigate further. Give the action an owner and make its scope explicit. A positive result on one page group does not automatically approve the same change across the entire catalogue.

If you keep it provisionally, record what would trigger another review. A different theme, a substantial change in product mix or newly discovered accessibility trouble may justify revisiting the decision. The log should make that future conversation easier.

Close the operational loop too. Remove abandoned test code, update editing instructions and make sure the live version matches the decision. Otherwise the document can say “reverted” while an overlooked component keeps serving the test variation.

Make the log useful next time

Store decisions somewhere the team already consults before proposing changes. Tag them by the customer problem or page area so an old delivery test can be found without remembering its project name. Link to the evidence rather than copying sensitive customer material into the summary.

Read past entries before repeating an idea. I’d rather spend ten minutes there than two weeks re-running a test. The earlier test may have answered the question, failed for a fixable reason or exposed a constraint that still exists. Reusing that learning is one of the cheapest improvements a store team can make.

Keep customer material out of the summary

Evidence can include support messages or session notes, but the decision log rarely needs personal details. Summarise the pattern and link to the approved private record for authorised reviewers. That makes the entry easier to share with the people responsible for future work.

Keep permissions in mind when linking dashboards too. A link nobody can open is a weak handover. Store an approved summary of the relevant observation, its date and its limitations so the decision remains understandable even if a reporting view changes later.

For your next experiment, create the decision entry before the first edit. If you cannot state the decision, the change and the evidence that would guide you, what exactly are you hoping the test will teach?