Section 3 · Establishing a baseline

The observe-set-validate workflow that turns test results into automated quality gates

Estimated time: 1 min

Scenario

A change passes functional tests and feels fine when you click through alone, but the p95 you care about was never written down. Under dozens of concurrent users, the same build might already be slow or noisy; you would not know until a release or an incident. A baseline turns “it felt OK” into numbers you can compare next week.

The baseline workflow

StepWhat you doWhat it produces
1. ObserveRun without thresholds at realistic loadActual p95, error rate, throughput
2. SetAdd thresholds based on observed values + headroomScript with pass/fail criteria
3. ValidateRe-run with thresholds, confirm consistencyWorking quality gate

The baseline methodology: observe, set, validate

Why this order matters

  • Observe before you set thresholds. Use measured values. Guesses drift and break trust in the gate.
  • Validate before you wire CI. Stable thresholds first. Flaky gates waste the whole team’s time.
  • Leave headroom on p95. A gap between what you saw and the limit absorbs normal jitter. It still catches real regressions.

The shift

Before baselinesAfter baselines
“The results looked okay to me”The test passed with exit code 0
Someone reviews metrics manuallyCI/CD decides automatically
Regressions found in productionRegressions blocked at merge