How to annotate a reporting change so future analysts understand it
The report jumps on Monday. Cue the celebrations, until somebody remembers the event definition changed over the weekend.
A reporting change can move the numbers without moving customer behaviour. Whoever reads this chart next year needs a note saying what changed, when, and which comparisons it affects. “Tracking updated” leaves them guessing.
I write the annotation while the change is fresh, with enough context to read the chart. You don’t need every technical detail on the dashboard. You need the boundary between two different ways of measuring.
State the change in business terms
Begin with what the report now counts differently. An enquiry event may have fired on a button click and now fires after a successful submission. The label looks the same, the meaning doesn’t.
Give the old definition and the new one, plus the affected pages, devices or channels. A change to one form isn’t a sitewide measurement repair, so don’t describe it as one.
Keep the language understandable to the people reading the report. They need to know whether a metric represents intention, completion or another observed action. Internal tag names can be linked from the detail without carrying the entire explanation.
Record the effective time accurately
Separate the implementation time from the first period with complete comparable data. A release can land midway through a reporting day, leaving that day with mixed collection. Mark it.
State the timezone used and any known reporting delay. Avoid an ambiguous date if colleagues work across locations. The note should let another analyst choose appropriate comparison windows without guessing when the new definition took effect.
If the exact boundary is uncertain, record the known interval and the evidence. Don’t invent a precise time because the annotation field asks for one. Honest uncertainty is part of reading the data.
Explain the comparability limit
Say which historical comparisons remain useful and which need qualification. A corrected duplicate event may reduce counts after the change. That does not establish that customer activity fell by the same amount.
Don’t quietly rewrite history to make the chart look continuous. If past data can’t be corrected reliably, keep it and explain why. I’d rather the reader understood the limit than got a tidy, misleading line.
I add a short example where the effect is easy to misread. A successful-submission definition leaves out failed attempts that the old click event counted. That shows colleagues why the two periods answer different questions.
Link the evidence and validation
Reference the change task, measurement plan and checks completed. Keep sensitive identifiers and customer information out of widely shared notes. The annotation can point an authorised maintainer to detail without exposing everything in the report.
Say how the new setup was verified. Did someone complete the journey and inspect the event and its values? A saved configuration isn’t proof the right action is collected.
Record known gaps. Some routes may still use the previous method or await another release. The annotation should not imply complete coverage when the rollout is partial. Give the remaining work an owner and a clear scope.
Put the explanation where analysts will find it
Use the reporting tool’s available annotation features where appropriate, and keep a durable change log in the team’s existing system. Link them when possible. Do not assume a note in a developer ticket will reach the person preparing a quarterly report.
Give the log a recognisable title and consistent fields. Metric, definition, effective period, scope, reason, validation and owner provide a useful starting point. Avoid adding so many fields that routine maintenance becomes burdensome.
Keep the main note short enough to read beside a chart. Put implementation detail in the linked record. An annotation should explain the interpretive consequence quickly while leaving a route to the evidence.
Review downstream reports
Check dashboards, scheduled exports and recurring presentations that use the affected measure. Their labels and commentary may need updating. A definition change can continue causing confusion if only the source dashboard contains the note.
Tell the report owners what action is needed. They may need to avoid a direct period comparison, update a calculation or explain the change to stakeholders. Do not make them infer the consequences from technical release notes.
Revisit the annotation when another measurement change occurs. Keep separate changes distinguishable rather than extending one generic “tracking updates” period indefinitely. A useful history lets the next analyst reconstruct which definition applied at each stage.
Distinguish a collection change from a reporting-only change. A new dashboard filter alters the displayed result while the underlying events stay the same. A tag repair changes what’s collected. Both need explaining, but they affect later analysis differently. Say which layer changed, whether the old view can be rebuilt, and who can answer questions. And name who maintains the annotation if the report moves to another tool.
Related reading
- Screaming Frog for small sites: the settings I change first
- A practical test for AI-written product descriptions
- The maintenance costs behind a small browser extension
Open a chart that changed after a tracking release. Could someone reading it next year explain the difference without asking whoever made the change?