Drift Monitor
Tell a real behavior change apart from ordinary sampling noise before you go chasing it.
- Sample workflow
- Exportable
- Runs in this browser
Runs in your browser. Nothing you enter is sent to a server, and nothing is stored unless you save it to this device.
Two snapshots in, a structured diff out
Paste or edit two system snapshots as JSON below, a baseline and a current one, each covering prompts, tools, permissions, knowledge sources, model settings, evaluation results, and any other metrics you track. Everything below is computed locally from exactly those two snapshots. This tool does not connect to any live system and does not monitor anything continuously; it compares two imports.
Awaiting input
Paste two snapshots, or start from a worked example
Edit the baseline and current snapshot JSON below by hand, or load one of two worked examples: one where a permission widened quietly, one where every change is explained.
Worked example
Each example is a complete, schema valid snapshot pair, not a fragment.
Baseline snapshot (JSON)
Schema version, prompts, tools, permissions, knowledge sources, model settings, evaluation, metrics, changelog, evaluationCoverage.
Current snapshot (JSON)
Its changelog entries are what mark a field path as an intentional change rather than unexplained drift.
Permission expansion, review this first
Findings
Every difference between the two snapshots, permission widenings first, then sorted by risk. Each one cites the exact field path it changed.
Idle
Missing evaluation coverage
Changed surfaces the current snapshot's evaluationCoverage list does not declare as tested.
Idle
Rollback checklist
Ordered to contain new access first, then correct configuration, then restore context. Capability reductions are listed last, since reverting those would mean granting access back.
Idle
Evaluation results, the one input with a significance test behind it
Everything above is a structural diff. This panel is the exception: it takes the
evaluation field of each snapshot, a plain pass count, and runs the same
rigorous two proportion significance engine regardless of what else changed.
The chance of a false alarm this test accepts, before any multiple comparisons correction.
How many metrics are being watched in the same sweep. Drives the Bonferroni correction below.
The detection rate the minimum detectable effect below is calculated for.
In percentage points. A verdict of insufficient data means this comparison could not reliably detect a change this size.
If the test cases themselves are different, no statistic below can separate that from a real behavior change.
Evaluation verdict
The plain language answer, and the one number it is actually based on.
Idle
Confidence interval on the difference
Current rate minus baseline rate. Whether this crosses zero is the actual answer to whether this is real.
Idle
Rates and Wilson intervals
Wilson score intervals, not the normal approximation, so these stay inside 0 to 100 percent even at extreme rates.
Idle
Significance tests
Both a large sample test and a small sample exact test, because small samples are the common case in eval work.
Idle
Statistical power
What this comparison could have detected, independent of what it actually found.
Idle
This is the minimum detectable effect: the smallest true difference these exact sample sizes could catch at the target power above. It does not depend on what was actually observed. This tool deliberately does not report "observed power", the chance of detecting the effect that was actually seen, because that number is a fixed rescaling of the p value already shown above and is considered poor practice for exactly that reason.
Worth checking before you trust this verdict
Run parameters
Every value below is stated, not hidden.
- Significance level (alpha)
- 0.05 from the significance level field below fixed
- Confidence level used per rate
- 95 percent from 1 minus alpha fixed
- Metrics monitored together
- 1 from the metrics monitored field below fixed
- Bonferroni corrected alpha
- 0.0500 from alpha divided by metrics monitored fixed
- Target power for detectable effect
- 80 percent from the target power field below fixed
- Smallest change said to matter
- 5.00 points from the meaningful effect field below fixed
What the significance panel assumes
- Independence: every test case counted in a sample is assumed independent of every other. This tool only receives aggregate counts, not per case results, so it cannot detect correlated failures (for example, five failures that all trace to one flaky dependency counted as five independent data points).
- Unpaired design: baseline and current are treated as two independent samples. If the identical fixed test cases were scored in both, the two outcomes per case are actually paired, and a paired test such as the McNemar test would use that structure and detect a smaller real change. Because only aggregate counts are available here, this tool cannot run that test; the independent samples test it does run is typically the more conservative choice in that situation, not the less trustworthy one.
- Stationarity: each sample is assumed to reflect one stable underlying rate for the whole window it was collected in, not a rate that itself drifted while that window was being measured.
- Identical eval set: the comparison is only meaningful if baseline and current were scored against the same evaluation set. A changed eval set produces a changed pass rate on its own, with no statistical test able to tell that apart from a real behavior change.
Note How this tool reads a snapshot pair
Every difference between the two snapshots becomes a finding: a category (prompt, tool, permission, knowledge source, model setting, or metric), a direction (added, removed, modified, or for permissions specifically, widened or narrowed), a risk level, and the exact field path that changed. A permission widening, a grant that got strictly bigger than the baseline, always sorts first, regardless of what other risk levels are in play elsewhere in the comparison. Narrowing a permission is the opposite event and is treated as good news, not a risk.
A change reads as intentional only when the current snapshot's own
changelog cites that exact field path with a reason. Anything changed
and not cited there is unexplained drift, which is the category that most needs a
human to look at it. Checking the box on a finding below edits that changelog for
you.
Wilson score intervals for each rate, not the normal approximation, because normal breaks at extreme rates. The interval on the difference is built from those same two Wilson intervals by Newcombe's hybrid method. A two proportion z test and an exact Fisher test both run; the tool says which one to trust, using the standard rule that every expected cell in the comparison table should be at least 5 for the normal approximation to hold. Monitoring more than one metric at once widens the interval the verdict reads, by the Bonferroni correction, rather than leaving it to a footnote.
Drift Monitor stopped responding.
Something in this tool threw an error. Your input was not sent anywhere, and nothing was saved. Reloading the page usually clears it.