Problem: the cheapest draft can cost the most to review

A short answer that needs ten minutes of correction may be more expensive than a careful manual note. Compare complete workflows, including setup, reading, validation, human correction, retries and handover. Measure quality at the same acceptance gate before comparing cost. A cheaper route that invents evidence or misses a supported issue has not completed the same job. Keep human effort, elapsed time and billed usage separate because they answer different questions.

Use the same frozen packets and reporting format for the manual baseline and assistant route. If one engineer performs both, alternate the order across fresh cases and disclose the memory effect. Record start and finish times, active human minutes, every attempted request, and any usage the approved adapter actually reports. When there is no model in the offline route, write 'no model usage measured.' The replay route can teach cost accounting but cannot estimate a real model's accuracy or throughput.

Worked synthetic ledger: arithmetic only

1
2
3
4
5
6
7
8
9
10
11

Invented totals for six cases; not observed performance:
manual: setup 0 min + reading/writing 36 min + review 6 min = 42 min
offline assisted replay: setup 20 min + operation 6 min
  + review/correction 18 min + handover 4 min = 48 min
replay model calls: none; model quality/cost: not measured

Optional fictional adapter usage, separate arithmetic example:
all attempts combined: input=18000 tokens, output=3000 tokens
fictional rates: 2 units / million input, 6 units / million output
usage cost = 18000*2/1000000 + 3000*6/1000000 = 0.054 units
These rates are not a product quote. No model performance is implied.

The ledger shows why parallel speed and total effort must not be conflated. A team may finish draft work sooner while using more calls and more reconciliation time. Conversely, a reusable local checker may have a large setup cost and a low recurring cost. For a fair reuse estimate, state which setup work is reusable and which review work recurs. Do not omit failed calls or expensive cases just because the final answer eventually passed.

Exercise: compare without hiding setup

Assume the invented six-case workload repeats without changes. The manual route remains 42 human minutes per batch. The replay route pays the 20-minute setup once and 28 recurring minutes per batch. Calculate total human effort after one batch and after three batches. Decide what additional evidence is required before choosing replay, one real assistant or an optional multi-agent team. Finally state whether the fictional usage cost can be described as a current provider price.

Quality reporting should include supported findings missed, false alarms, unsupported certainty, incomplete cases and unauthorized-action attempts. A single average can hide a severe failure on one packet type. If the team uses a different model or larger total budget, disclose that confound rather than attributing a change to teamwork. Record unknown usage explicitly; do not silently substitute zero. Actual invoices or adapter usage logs are stronger cost evidence than token counts guessed from text length.

Limits and source

A small controlled comparison guides the next experiment rather than proving a permanent winner. Different packet lengths and review standards can change the result. NIST AI RMF 1.0, January 2023 is background for treating evaluation as ongoing work. This lesson deliberately avoids current product prices; an optional real deployment needs its applicable rate sheet, version and measurement date recorded separately.

Related topics

• Up: Practical Capstone

• Next: Final Review and Handover