Goal and protocol revision log
You tightened the criterion in January because everyone had drifted. Sensible. It also means the autumn data and the spring data are measuring different things, and in June nobody will remember that happened.
Why a log rather than a note in the margin
A definition change is invisible in the data. The numbers keep the same shape, the same units and the same column headings, and a chart will happily draw a line straight through the moment the measurement changed underneath it.
The only defence is writing it down when it happens. Six months later the change is reconstructed from memory, if at all, and usually while someone is waiting for an answer.
What counts as a revision
- Always log it
- The criterion changes. The opportunity definition changes. Prompted responses start or stop counting. The measurement method changes. The unit or denominator changes.
- Usually log it
- The setting or activity changes. The observation window changes. A new person starts collecting. The collection schedule changes materially.
- No need
- You bought different paper. You moved the folder. You started using an app to record the same measurement the same way.
The test: would data collected after this change be directly comparable with data collected before it? If you hesitate, log it.
The log
Goal and protocol revision log
| Date | Version | What changed | Why | Changed by | Comparable with earlier data? |
|---|---|---|---|---|---|
- Yes
- Wording tidied, measurement unchanged. Report as one series.
- No
- What is being counted has changed. Report the periods separately and say why.
- Partly
- Comparable for some purposes only. Write which, here, not in an email.
Worked example
Fictional, and the ordinary case rather than a dramatic one.
| Date | Version | What changed | Why | Comparable? |
|---|---|---|---|---|
| 12 Sep | v1 | Goal written. Opportunity is each two-step direction during arrival. Met criterion is beginning within 10 seconds, no prompt. | Annual review | — |
| 14 Nov | v1.1 | Added that a repeated direction is still one opportunity, not two. | Two collectors were counting repeats differently; found in a monthly check | Partly. Counts from before this are slightly inflated on the denominator for one collector. |
| 9 Jan | v2 | Met criterion narrowed: a gestural prompt no longer counts as met. | Team decided the goal was about independence and the data was not reflecting it | No. Autumn and spring are different measurements. Report separately. |
| 3 Mar | v2 | Collection moved from arrival to the transition after lunch. | Arrival routine changed when the timetable did | Partly. Same criterion, different context. Note it on the graph. |
In June, the 9 January row is what stops a well-intentioned person putting the whole year on one chart and reporting a decline the student did not have.
Reporting across a revision
- Report each version’s data separately, each with its own denominator.
- Say what changed and when, in the report, in a sentence.
- Do not average across the change to produce one tidy number.
- If the chart spans the change, mark the date on it.
- Keep the pre-change data. It is still evidence about what happened before.