Evaluation Workbench
Design an evaluation that would actually catch the failure you are worried about.
- Sample workflow
- Exportable
- Runs in this browser
Runs in your browser. Nothing you enter is sent to a server, and nothing is stored unless you save it to this device.
Evaluation set
Name what this set is for, and which concerns it should cover.
Idle
Declared up front. The coverage report below checks whether any case actually exercises each one.
Rubric and thresholds
Both rubrics are fully supported. A deterministic check always resolves to pass or fail regardless of which rubric is in force.
Idle
Fraction of fully scored cases a candidate must pass, on top of the critical case rule.
Candidates
Two or more configurations to compare. Paste each one's output per case below.
Idle
Cases
Cases are the primary object. Each has an input, one or more expected properties, and a critical flag.
Idle
Awaiting input
No cases yet
Add a case above, or load the sample to see a complete scored evaluation set end to end.
Candidate results
Computed only from the checks and scores entered above. A critical case failure overrides the pass rate.
Idle
Coverage gaps
Declared concerns no case actually exercises, cases that check nothing, and whether there are enough candidates to compare.
Idle
Disagreement
Where candidates diverge most on the same case, and where the same concern scored inconsistently for one candidate.
Idle
Detectable regression
Given the number of cases in this set, the smallest true regression this eval could catch. A secondary, honest estimate, not the product.
Idle
Accepts the JSON this tool exports.
Note Why can a high average still be a failing eval
An average is a summary, and a summary is exactly what a serious failure can hide inside. A candidate that gets nine of ten cases right looks like a strong candidate right up until the tenth case is the one where it leaks a credential or approves a transaction it should have refused. Averaging that case in with the rest treats a rare, severe failure the same as an ordinary miss.
Marking a case critical says plainly that it is not the same as the rest. This tool checks every critical case on its own terms, pass or fail, with no partial credit and no averaging. If one fails, the candidate fails, and the pass rate above is shown next to that result rather than in place of it.
Evaluation Workbench stopped responding.
Something in this tool threw an error. Your input was not sent anywhere, and nothing was saved. Reloading the page usually clears it.