Forward-Deployed Engineering
Real cases with known answers are agreed with the people who do the work, and every release is scored against them before it ships. After cut-over the same score runs on live traffic, with a threshold that pages someone.
The method
Each part exists before the next is built. Together they turn the question of whether the system is right into a number both sides can read.
Real cases from your records with the correct answer written beside each, chosen with the people who do the work. It covers the ordinary cases, the exceptions they know about and the ones that went wrong last year. It is agreed before anything is scored, and it grows every time a new kind of case appears.
Code in your repository that runs a candidate over the golden set and reports precision, recall, latency and cost per case. It runs the same way on a laptop and in your pipeline, so a score means the same thing wherever it was produced.
Every pull request that touches the workflow is scored before it is merged. A score below the last release fails the build. The comparison is against the previous release, not against a demonstration, so a quiet decline is caught the week it happens.
A release threshold decides what may ship; an operating threshold decides when someone is paged. Both are written down at scoping, with the person who set them and the reason. The same score runs on live traffic after cut-over, sampled and reviewed, so the number you release on is the number you run on.
The difference
The model in both is often the same. What differs is whether anyone can say, with a number, how often it is right, and whether they would know when that changes.
BEFORE
AFTER
What you keep
Evaluation runs inside every engagement. These are the parts of it you hold when the engineer leaves, checked off together at handover.
EVALUATION MANIFEST
Real cases with known answers, versioned in your repository, with a record of who agreed each one and when.
Code that scores a candidate on the golden set and reports precision, recall, latency and cost per case, runnable by anyone with the repository.
Wired into your pipeline so a score below the last release fails the build before a merge.
The release threshold and the operating threshold, written down with the person who set them and why.
Live score, volume, latency, cost and the queue of cases handed to a person, in your monitoring, under your accounts.
A written routine for re-scoring on fresh cases as your records change, and for extending the golden set when a new kind of case appears.
Specified
The terms as they stand inside every engagement. The workflow moves the detail; it does not move the shape.
Fit
Questions
Big enough to cover the kinds of case the workflow meets, including the exceptions the people who do the work know about. The number is set by the workflow, not by a rule. It grows whenever a new kind of case appears, and the growth is recorded.
The people who do the work today, with the owner of the outcome settling disagreements. The engineer records the answer and the reason; the engineer does not decide it.
The cut-over waits. The score is the argument for shipping; without it there is no argument. The cases that fail are looked at one by one, and the fix or the threshold is changed in writing.
The regression run adds minutes to a pull request, not days. It runs where your tests run. The alternative is finding the regression from a user, which costs more than the minutes.
The same score, on a sample of live traffic, plus volume, latency, cost and the queue of cases handed to a person. An operating threshold decides when someone is paged, and that person is named on the rota.
Yes. The golden set and the harness do not depend on who wrote the system. It is scoped on its own: the golden set agreed, the harness written into your pipeline, the thresholds set and the dashboards live, then handed over.
Forward-Deployed Engineering
Start
Thirty minutes with an engineer is enough to say what the golden set would need to cover and where the harness would run.