heurema
A smooth demo proves that a path can run once. It says little about accepted results, review load, failures, or recovery. Wider authority needs an operating history.
I look for evidence in four groups. The exact thresholds depend on the process, so the review should record them before the pilot starts.
Outcome
Start with the result that justified the experiment. Useful measures include time to an accepted result, queue size, response time, completed cases, or hours returned to the responsible person.
Compare similar work with a real baseline. Faster handling of simple cases can hide a growing queue of exceptions.
Quality
Track first-pass acceptance, rework, escaped errors, and the severity of each miss. Separate a formatting correction from a mistake that changes policy, data, money, or production behavior.
The useful unit is an accepted result. Generated drafts and completed runs belong in the diagnostic log.
Control
Check whether required tests ran, approvals happened at the right point, sources were recorded, and stop conditions worked. A control that people routinely bypass provides weak protection.
Measure the burden on people as well. Review time, approval load, exception rate, and repeated clarification show whether the workflow is usable in ordinary work.
Recovery belongs in this group. The team should know how long it takes to disable the workflow, restore the previous state, and explain the event from the retained evidence.
Economics
Count model use, tool calls, retries, review time, and rework. Cost per run is easy to calculate and often misleading. Cost per accepted result describes the work the team can actually keep.
A cheaper model can cost more after retries. A stronger model still performs poorly inside an unclear process. Measure the complete loop.
Pair every speed metric with a guardrail
- cycle time with escaped errors
- accepted volume with rework
- autonomous actions with exception severity
- model cost with review time
- approval speed with policy violations
- completed tasks with trace completeness
Paired metrics make local optimization harder. They also show where the next change belongs, whether in the model, context, tools, checks, or process.
Make one explicit decision
At the end of the pilot, choose one action.
- Expand one boundary.
- Hold the current boundary and collect more evidence.
- Narrow or redesign the work.
- Stop the experiment.
An expansion note should name the evidence, the new permission, the limits that remain, the rollback path, and the condition that reopens the decision. Change one material boundary at a time. Otherwise the next result will not explain which change mattered.