Latency Budgeter
Find out which stage of your pipeline spends the most user patience, before you ship it.
- Sample workflow
- Exportable
- Runs in this browser
Runs in your browser. Nothing you enter is sent anywhere.
Baseline pipeline
Proposed pipeline
Awaiting input
No pipeline yet
Describe the stages a request passes through, from client to reply, or load the sample RAG pipeline to see the tool work end to end.
Composed total versus the naive sum
The naive sum adds every stage's own p99 together, the common mistake this tool exists to correct. The composed total is the honest figure.
Idle
Baseline versus proposed
The delta at p50 and at p99, and which stage produced the most of it.
Idle
Time to first token versus time to last token
Only differs when a stage in the pipeline streams. A streaming UI changes perceived latency completely.
Idle
Critical path
Every stage in order. A parallel group is priced at its slowest member, marked as the critical stage; the others in that group carry slack.
Idle
Dominant stage and the halving test
The stage contributing the most to the total, and what happens to the total if its latency were cut in half. Amdahl reasoning: a stage sharing a parallel group with a similarly slow neighbor cannot be helped past that neighbor.
Idle
Budget verdict
Whether each pipeline fits the stated user facing budget, at p50 and at p99, for the full reply and, when it applies, for the first token.
Idle
Every formula, inspectable
PRD acceptance criterion: formulas are inspectable. Every stage and every group that fed into the totals above, with its own arithmetic.
Idle
Note Show the arithmetic for the baseline pipeline
Note Show the arithmetic for the proposed pipeline
Modeling assumptions
Every value below is stated, not hidden.
- Percentile shape fit to every stage
- Lognormal, from p50 and p99 from method fixed
- Series stages combine by
- Summing means and variances, then a refit from method fixed
- Parallel stages combine by
- The distribution of the maximum, solved directly from method fixed
- Retry attempts allowed per stage
- 1 to 5 from method fixed
- Concurrency model
- Queueing multiplier, 1 / (1 minus utilization) from method fixed
- Maximum utilization allowed per stage
- 97 percent from method fixed
- Default user facing budget
- 2000 ms from your input
- Independence between stages
- Assumed. See the warning below. from method fixed
The independence assumption is the one most likely to be false
Every formula above assumes each stage is independent of the others. That assumption is the one most likely to be false in a real system: a GPU queue backing up under load, a shared rate limit, or a connection pool exhausted by every stage at once all make delays correlate instead of behaving independently. When a shared resource saturates, the true p99 is worse than this tool reports, and the gap is largest exactly when it is least affordable to be wrong about it.
Note Why do not p99 latencies just add up?
Each stage is fit to a lognormal curve from its own p50 and p99. Stages in series combine by summing means (always exact) and summing variances (exact if the stages are independent), then a lognormal is refit to that combined mean and variance to read off a composed p50 and p99. Parallel stages do not sum: a parallel group finishes when its slowest member finishes, so its p50 and p99 come from the distribution of the maximum of its members, solved directly rather than sampled. No Monte Carlo runs anywhere in this tool, so the same input always produces the same output.
This tool never samples a random number to answer that question. Every composed figure comes from closed form curve fitting and, for parallel groups, bisection against the curves' own CDFs. The same pipeline always produces the same numbers.
Latency Budgeter stopped responding.
Something in this tool threw an error. Your input was not sent anywhere, and nothing was saved. Reloading the page usually clears it.