Forward-Deployed Engineering07
A regression is caughtby the harness,not by a user.
Real cases with known answers are agreed with the people who do the work, and every release is scored against them before it ships. After cut-over the same score runs on live traffic, with a threshold that pages someone.
- r3 · recall on new case type · golden set extended
- r5 · precision under threshold · release held
- r6 · scored above threshold · released
- Golden set
- Agreed with operations by week two
- Scored
- Every release, before it ships
- Threshold
- Written down; pages someone
- Lives in
- Your pipeline, your dashboards
FDE / 01The method
Four parts, in order.
Each part exists before the next is built. Together they turn the question of whether the system is right into a number both sides can read.
The golden set
Real cases from your records with the correct answer written beside each, chosen with the people who do the work. It covers the ordinary cases, the exceptions they know about and the ones that went wrong last year. It is agreed before anything is scored, and it grows every time a new kind of case appears.
The harness
Code in your repository that runs a candidate over the golden set and reports precision, recall, latency and cost per case. It runs the same way on a laptop and in your pipeline, so a score means the same thing wherever it was produced.
The regression run
Every pull request that touches the workflow is scored before it is merged. A score below the last release fails the build. The comparison is against the previous release, not against a demonstration, so a quiet decline is caught the week it happens.
The thresholds
A release threshold decides what may ship; an operating threshold decides when someone is paged. Both are written down at scoping, with the person who set them and the reason. The same score runs on live traffic after cut-over, sampled and reviewed, so the number you release on is the number you run on.
FDE / 02The difference
A demonstration and a release.
The model in both is often the same. What differs is whether anyone can say, with a number, how often it is right, and whether they would know when that changes.
A demo that impressed
- Ran on a handful of cases somebody picked because they worked
- Judged by the room: it looked right
- No one can say what its precision is
- A prompt change last month made it worse; nobody noticed
- Whether it is safe to ship is a matter of opinion
A release that was scored
- Scored on a golden set operations chose, exceptions included
- Judged by the harness: precision, recall, latency, cost per case
- The score is in the pull request, next to the last release's
- A drop below threshold failed the build and held the release
- Whether it is safe to ship is a number, and it is written down
FDE / 03What you keep
6 ITEMS
Six things, all in your accounts.
Evaluation runs inside every engagement. These are the parts of it you hold when the engineer leaves, checked off together at handover.
- The golden set
Real cases with known answers, versioned in your repository, with a record of who agreed each one and when.
- The scoring harness
Code that scores a candidate on the golden set and reports precision, recall, latency and cost per case, runnable by anyone with the repository.
- The regression run
Wired into your pipeline so a score below the last release fails the build before a merge.
- The thresholds
The release threshold and the operating threshold, written down with the person who set them and why.
- The dashboards
Live score, volume, latency, cost and the queue of cases handed to a person, in your monitoring, under your accounts.
- The drift review
A written routine for re-scoring on fresh cases as your records change, and for extending the golden set when a new kind of case appears.
FDE / 04Specified
10 FIELDS
Evaluation, specified.
The terms as they stand inside every engagement. The workflow moves the detail; it does not move the shape.
- Shape
- Part of every engagement; also scoped on its own for a system already in production
- Golden set
- Real cases with known answers, chosen with the people who do the work, agreed by the end of week two
- Who agrees it
- The owner of the outcome and the people who do the work; not the engineer alone
- What is measured
- Precision, recall, latency and cost per case, plus whatever the scoped metric requires
- When it runs
- On every pull request that touches the workflow, and on a sample of live traffic after cut-over
- What fails a build
- A score below the previous release on the golden set
- What pages someone
- The operating threshold, set in writing at scoping, with a named person on the rota
- Where it lives
- Your repository, your pipeline, your monitoring; nothing hosted by us
- Drift
- Re-scored on fresh cases at an agreed interval; the golden set extended when a new kind of case appears
- At handover
- Your engineers run the harness, read the dashboards and change a threshold without us in the room
FDE / 05Fit
When to scope it alone
Scope evaluation on its own when
- A system is already in production and nobody can say how often it is right
- A prompt or model change went out and quality moved, and no one knows which way
- The vendor's own benchmark is the only number you have
- A regulator or an auditor has asked how the system is tested
- Two candidates are being compared by opinion rather than a score
It is already included when
- The workflow is being built in a Production Sprint; the golden set is a week-two deliverable
- A pilot is being hardened; the score is one of the go-live criteria
- The system is under Managed Operations; re-scoring is part of the monthly report
- A pod is in your repository; the regression run is part of the pipeline it keeps
- Handover is under way; running the harness is part of the Handover & Enablement dry run
FDE / 06Questions
6 QUESTIONS
Asked about evaluation.
How big does the golden set have to be?
Big enough to cover the kinds of case the workflow meets, including the exceptions the people who do the work know about. The number is set by the workflow, not by a rule. It grows whenever a new kind of case appears, and the growth is recorded.
Who decides what the right answer is?
The people who do the work today, with the owner of the outcome settling disagreements. The engineer records the answer and the reason; the engineer does not decide it.
What if the score is below threshold at cut-over?
The cut-over waits. The score is the argument for shipping; without it there is no argument. The cases that fail are looked at one by one, and the fix or the threshold is changed in writing.
Does evaluation slow the pipeline down?
The regression run adds minutes to a pull request, not days. It runs where your tests run. The alternative is finding the regression from a user, which costs more than the minutes.
What is watched after cut-over?
The same score, on a sample of live traffic, plus volume, latency, cost and the queue of cases handed to a person. An operating threshold decides when someone is paged, and that person is named on the rota.
Can we run this on a system somebody else built?
Yes. The golden set and the harness do not depend on who wrote the system. It is scoped on its own: the golden set agreed, the harness written into your pipeline, the thresholds set and the dashboards live, then handed over.
FDE / ENDStart
Ship on a number,not on an impression.
Thirty minutes with an engineer is enough to say what the golden set would need to cover and where the harness would run.